Business Case:
Our client, a multinational technology company, used to store their data via on-premises servers and Azure VMs. This storage system resulted in significant costs, slow processing times, and difficulty consolidating infrastructure.
On average, our client’s data required 24 hours to pull and process. Their existing solution to slow processing times was simply to upgrade servers, which resulted in a significant infrastructure cost. It was also difficult to patch, manage, and maintain infrastructure. Our client wanted to resolve the many server dependencies across multiple streams.
Functional data systems are crucial to business success, generating key insights that drive sales and marketing. Our client deals with terabytes, and even petabytes, of data. They needed assistance developing a cloud migration strategy that accounted for all of their business and functional needs.
Key Challenges:
•
Enable real-time reporting
•
Facilitate independent refreshes of multiple streams
•
Easily manage and maintain infrastructure
•
Decrease costs
Our Solution:
We developed a cloud migration strategy for our client, helping them move their data from on-prem servers to an Azure Data Warehouse system.
 |
| Table 1: Feature Breakdown On-prem Versus Cloud Architecture |
To determine which technology would be most relevant to our client, we developed multiple proofs of concept. We analyzed the following technologies:
•
Azure Analysis Services: We looked into using Azure Analysis Services to optimize processing through Azure Data Warehouse, Azure SQL Database, and Azure Data Lake Services. We examined how multi-partitioning and file splitting affected optimization.
•
Azure Data Warehouse: We processed multiple models simultaneously by cloning Azure Data Warehouse to parallelize processing and improve execution speed.
•
Azure SQL Database: We conducted a cost-benefit analysis for Azure SQL Database in comparison to Azure Data Warehouse.
Since our client’s data system transfers 50 terabytes of data a day, we chose to go with Azure Data Warehouse. Azure Data Warehouse offers more scalability when compared to Azure SQL Database.
Once we knew which technology best suited our client, we moved onto implementation. We used Azure Data Lake Storage for storage, Azure Databricks for processing, tabular models and SQL for publishing, and Power BI and Excel for visualization.
Find out more about the steps to our cloud migration strategy on our blog.
Business Outcomes:
We leveraged the serverless, scalable, and distributed architecture of the cloud to ease our client’s infrastructure maintenance. We enabled our client to engage in real-time reporting via a real-time data refresh pipeline. We enabled data publishes based on source availability using an intelligent job monitoring and validation framework. Our solution reduced our client’s data latency, resulting in a refresh cycle that was twice as efficient.
Highlights:
• We migrated our client from an on-prem server to the cloud, accelerating data processing and real-time reporting while reducing maintenance costs
• Our solution resulted in a data refresh cycle that was twice as efficient
• A cloud infrastructure offered our client unlimited scalability