Customer overview
Our customer, a U.S.-based grocery retailer specializing in fresh, natural, and organic products, maintains a strong presence nationwide and is committed to delivering health-conscious, personalized shopping experiences through digital transformation and data-driven initiatives.
Business challenge
Our customer aimed to elevate their customer experience through highly personalized engagement but faced significant hurdles.
- Legacy data infrastructure could not scale to meet the demands of efficient machine learning model training and deployment.
- Disconnected systems and ungoverned data processes led to inconsistent reporting and limited the ability to automate features or generate reliable insights across departments.
- Without a unified and secure MLOps framework, integrating, monitoring, and governing ML models in production proved challenging—slowing personalization efforts and increasing operational risk.
Solution overview
To enhance their customer experience and deliver greater value, our customer set out to build intelligent models capable of predicting preferences, recommending personalized actions, and identifying the next best step for every shopper.
They implemented a modern, enterprise-grade MLOps CI/CD framework built on Databricks Asset Bundles (DAB), MLflow, and Azure DevOps. This unified and automated pipeline streamlined every stage of the machine learning lifecycle, from model development and orchestration to deployment and monitoring.
By automating and standardizing model workflows, our customer was able to accelerate the transition from experimentation to production with greater speed, reliability, and governance. The framework now serves as a scalable foundation for future ML initiatives and stands as a benchmark for operational excellence within the data science practice.
Key components of the solution:
- Code modularization for reusability and maintainability.
- Databricks workflow orchestration using Databricks Asset Bundle for environment-specific deployments.
- Automated unit testing and granular logging at the module level.
- CI/CD pipeline setup using Azure DevOps with controlled release approvals and branching strategies.
Implementation details
CI/CD Architecture using Databricks Asset Bundle:
Development stage:
-
Code modularization:
- Achieved code reusability by leveraging Object Oriented and functional programming methodology by reusing python functions as modules and creating init.py files in directory and appending the root directory to sys path.
-
Databricks notebooks/workflow orchestration using Databricks Asset Bundle:
-
Modular and declarative job definitions
- Defined ML workflows (training, evaluation, batch scoring) in modular YAML based Asset Bundles.
- Simplified orchestration by codifying dependencies between jobs (e.g., training must complete before deployment).
-
Environment-specific configurations
- Used DAB environment support to manage differences in configurations (like cluster, parameters, permissions) across dev, staging, and production environments.
- Eliminated manual job configuration drift between environments.
-
Seamless CI/CD integration
- Integrated DAB with Azure DevOps pipelines to enable automated deployment of ML workflows.
- Triggered model training, validation, and registration as part of a single pipeline using version-controlled bundles.
-
Version control and auditability
- Stored Asset Bundles in Git for full version control, allowing rollback, auditing, and traceability of ML pipeline changes.
- Enabled reproducibility of workflows by pinning specific bundle versions.
-
Automated CI unit testing
- All code changes are subject to automated unit testing before they can be merged into the main branch. This ensures a baseline level of quality and reduces the risk of issues during deployment.
-
Deployment stage:
We designed a two-stage deployment pipeline using Databricks Asset Bundles to handle distinct ML and feature engineering workloads:
-
Data science stage
- Deploys all schemas except feature automation.
- Flexible selection of specific schemas is possible via the deployment folder variable; if left blank, all eligible schemas (excluding feature automation) are deployed.
-
Feature automation stage
- Manages deployment of all feature automation models.
- All bundles are deployed unless specific models are mentioned in deployment folder.
-
Selective deployment
- Users can enter comma-separated folder names to deploy only required schemas or models.
-
Default full deployment
- Leaving deployment folder empty, deploys all relevant bundles by default.
-
Manual triggers
- Stages require manual initiation, offering controlled rollouts.
Results and benefits
- Declarative infrastructure: Define jobs, clusters, permissions, and dependencies in code (as YAML), ensuring reproducible deployments.
- Bundle once, deploy anywhere: Use the same bundle across multiple environments by simply switching context and config.
- Databricks asset bundle: Enables consistent and repeatable model deployments across development, pre-prod, and production environments with minimal manual intervention.
- Faster time-to-production: Reduced manual effort in releasing ML assets, accelerating personalization feature releases.
- Consistent deployments: Simplified and standardized deployments across dev, test, and prod environments.
- DAB permissions: Permissions for notebooks, workflows, jobs, and model objects were defined and enforced through DAB configurations, ensuring only authorized users could access or modify critical components.
- Cluster governance: Cluster policies and permissions were enforced to control which users or services could run specific workflows on designated compute resources, enhancing security and cost management.
Contact us
Are you ready to empower your teams with seamless analytics?
Send us an email at CustomerSuccess@MAQSoftware.com.