Data Lake and MLOps Solution
Executive Summary
A unified data lake and MLOps platform built for a Hospitality group, integrating guest data from property management systems, booking engines, and loyalty programs into a governed data lake on AWS, with automated ML pipelines for demand forecasting, dynamic pricing, and guest personalization models.
Business Problem
The Hospitality group operated 50+ properties with data fragmented across property management systems, central reservation systems, point-of-sale terminals, and a loyalty program database. Revenue management teams relied on spreadsheet-based forecasting with 2-week-old data, missing short-term demand signals that competitors captured for dynamic pricing. Guest personalization was limited to basic loyalty tier segmentation, ignoring rich behavioral data from on-property interactions. Data extraction for analytics required manual SFTP transfers with inconsistent schedules, and no ML infrastructure existed for the revenue science team to deploy predictive models.
Architecture Overview
The solution implements a serverless data lake architecture with automated MLOps pipelines. Guest and operational data is collected from source systems via SFTP and API integrations into an S3-based data lake organized by domain (reservations, guest profiles, revenue, operations). AWS Glue handles ETL transformations with data catalog management for discoverability. SageMaker provides the ML platform for model development, training, and deployment. Lambda functions and EventBridge orchestrate pipeline automation including scheduled retraining, model evaluation, and deployment workflows. GitLab CI/CD manages infrastructure-as-code and model code versioning.
Solution Design
Source system data arrives via scheduled SFTP transfers and API polling orchestrated by EventBridge rules. Landing zone S3 buckets receive raw data with event notifications triggering Glue ETL jobs for schema validation, deduplication, and transformation into analytics-ready formats. The Glue Data Catalog provides a searchable metadata layer enabling self-service data discovery. SageMaker notebooks provide an experimentation environment with managed infrastructure for the revenue science team. Training pipelines execute on SageMaker managed training instances with hyperparameter tuning jobs for demand forecasting and pricing models. Trained models are deployed to SageMaker endpoints with auto-scaling for real-time inference. Lambda functions implement the scoring pipeline for batch predictions (weekly demand forecasts) and real-time predictions (dynamic pricing adjustments). EventBridge schedules trigger automated model retraining when new booking data accumulates past configured thresholds. GitLab pipelines enforce code review, testing, and staged deployment for both infrastructure and model code.
Architecture Diagrams
Technology Stack
Security
Guest personally identifiable information is protected through column-level encryption in the data lake with access policies enforced via AWS Lake Formation. IAM roles implement least-privilege access with separate permissions for data engineering, data science, and production model serving workloads. S3 bucket policies prevent cross-account access, and VPC endpoints ensure data transfer between services never traverses the public internet. SageMaker notebooks operate in isolated VPCs with no internet egress, accessing only approved data sources. Guest loyalty data access requires explicit approval workflows with audit logging of all PII access events. GDPR-compliant data retention policies are enforced through S3 lifecycle rules with automated deletion of guest data past retention windows.
Scalability
The data lake ingests data from 50+ properties generating 10 million guest interaction events daily. Glue ETL jobs scale automatically with data volume using DPU auto-scaling. SageMaker training instances are provisioned on-demand and released after training completion, eliminating idle infrastructure costs. Lambda-based scoring pipelines handle burst inference requests during peak booking periods without pre-provisioning. S3 provides unlimited storage scaling for historical data retention supporting multi-year trend analysis. The architecture supports onboarding additional properties through configuration changes to the ingestion layer without infrastructure modifications.
Performance Metrics
- Demand forecast accuracy improved from 62% to 87% through ML models trained on unified guest and booking data
- Revenue per available room (RevPAR) increased by 14% through dynamic pricing powered by real-time demand signals
- Data freshness improved from 2-week lag to daily availability for analytics and model training
- Model retraining cycle automated at weekly cadence, keeping predictions current with booking trends
- Data scientist time-to-first-model reduced from 4 months to 3 weeks with managed SageMaker infrastructure
- ETL pipeline processing completes within 2-hour nightly window for all 50+ property data sources
- 99.5% model serving availability for real-time dynamic pricing endpoints
Challenges
- Property management systems across the hotel group used different data formats and schemas, requiring a flexible ingestion framework with property-specific transformation configurations to normalize data into a unified schema
- Guest identity resolution across systems was complex — the same guest could appear with different identifiers in the PMS, booking engine, and loyalty system, requiring fuzzy matching and deterministic linking logic
- Demand forecasting models needed to account for highly seasonal patterns, local events, and economic factors that varied significantly across property locations and segments
- SFTP-based data collection from legacy property systems introduced latency and reliability challenges compared to API-based integrations, requiring robust retry and reconciliation mechanisms
- Revenue management teams had established manual processes and were initially resistant to ML-driven pricing recommendations, requiring a gradual trust-building approach with explainable model outputs
Lessons Learned
- Starting with a simple demand forecasting model that outperformed the existing spreadsheet approach built organizational trust in ML-driven recommendations — complexity was added incrementally as confidence grew
- Glue Data Catalog with comprehensive metadata tagging became the most valued capability for the analytics team, enabling self-service data discovery that previously required data engineering support for every query
- EventBridge-driven orchestration with Lambda provided the right balance of automation and visibility — pipeline failures are immediately surfaced with contextual information for rapid resolution
- Guest identity resolution was the highest-value data engineering investment — unified guest profiles unlocked cross-property personalization that individual property data could never achieve
- SageMaker managed training eliminated infrastructure management overhead that had previously consumed 40% of data scientist time, allowing focus on model innovation rather than DevOps
Business Outcomes
- RevPAR increased by 14% across the property portfolio through ML-powered dynamic pricing, generating $8.5M additional annual revenue
- Demand forecast accuracy improvement from 62% to 87% enabled better staffing and inventory decisions, reducing operational waste by 22%
- Guest personalization driven by unified profiles increased loyalty program engagement by 28%
- Time-to-insight for business analysts reduced from weeks to hours through self-service data access
- Data science team scaled from 2 to 8 members without proportional infrastructure engineering growth due to managed platform capabilities
- Cross-property guest insights enabled targeted marketing campaigns with 3x higher conversion rates versus property-level segmentation
Questions
FAQ
How does the platform handle new property onboarding?
New properties are onboarded through a configuration-driven process that defines the source system type, connection parameters, schema mapping rules, and transformation logic. Standard templates exist for common PMS platforms, reducing onboarding from weeks to days. Initial model training uses transfer learning from similar property segments while local data accumulates for property-specific fine-tuning.
How are pricing recommendations presented to revenue managers?
The dynamic pricing model generates recommendations with confidence intervals and explainability factors (demand drivers, competitive positioning, historical patterns). Revenue managers review recommendations through a dashboard that highlights deviations from current pricing with business context. They can accept, modify, or override recommendations, with overrides feeding back as training signal for model improvement.
What data governance controls exist for guest PII?
AWS Lake Formation enforces column-level access controls on PII fields. Data consumers access de-identified datasets by default, with PII access requiring explicit approval workflows with business justification. All PII access is audit-logged, and automated scans detect any PII leakage into unauthorized locations. GDPR data subject access requests are handled through automated pipelines that locate and export all data associated with a guest identifier.
Future Roadmap
Planned enhancements include real-time streaming ingestion from IoT sensors in properties for occupancy and energy optimization, computer vision integration for guest experience monitoring in public spaces, natural language interfaces for business users to query the data lake conversationally, and expansion to support restaurant and spa revenue optimization models.