Business Case:
Our client, the sales strategy team at a Fortune 500 company, operates on a global scale. Sales strategists base critical business decisions on new and historical sales data. In our client’s fast-paced environment, sales strategists need a data management system that supports near real-time (NRT) processing. However, high performance requirements can lead to excessive costs and increased platform complexity.
Our client previously stored and organized data on-premises, which caused downstream data refresh lags. With high refresh times, deriving real-time insights from business intelligence (BI) systems such as Power BI proved difficult. The current system was also difficult to scale, unable to keep up with increasing volumes of data. Identifying an optimization opportunity, we approached the client with a proposal: migrate data management to the cloud. Cloud systems offer many live data and analytic capabilities. Seeking to reduce data latency, our client partnered with us to migrate their entire data system to the cloud.
Key Challenges:
•
Migrate the client’s data management system to the cloud
•
Reduce data latency and increase performance of data management system
•
Enable near real-time reporting in our client’s BI environment
•
Implement a scalable, cost-efficient data management system
Our Solution:
We migrated our client’s on-prem system to an Azure-based data management system.
 |
| Figure 1: Solution Design |
We implemented Azure technology at all levels of our client’s data management system. Our solution can be divided into four stages. The first stage is the source, where data is stored. The second stage is extraction, where a set of Azure Data Factory (ADF) pipelines pull data from our client’s data sources. The third stage is incubation, where data is processed with Azure Databricks (ADB) and stored with Azure Data Lake Storage (ADLS). The final stage is reporting, which is built using Azure Analysis services supported for Power BI reporting.
 |
| Table 1: System Comparison |
Business Outcomes:
In migrating our client’s data management system to a cloud-based Azure system, we drastically increased data processing speed. We reduced processing time for 0.7 million rows of data from half an hour to two minutes. Azure data management can respond to both large quantities and sizes of data. Our solution processes 2.3 million rows in just two minutes and reduced the overall processing time by 66%.
The cloud offers greater flexibility and scalability to our client’s 2.5K sales strategists. Previously, our client took manual snapshots of their weekly sales data to track key performance indicators over time. Our solution automates snapshots with a configurable, built-in ADB feature. Compared to on-prem SQL systems, ADLS is easy to scale, capable of storing thousands of GB of data. Azure offers automated system start, stop, and scaling, so the client’s system can grow in line with their data. Our solution saves the client $5-6K a month in data management and processing costs.
Migrating to Azure enables our client to continuously improve their analytics with artificial intelligence (AI) and machine learning (ML). Our client can easily implement out-of-the-box Azure capabilities such as text analytics, media analytics, and cognitive services. The platform is optimized for near real-time processing, enabling sales strategists to execute initiatives with up-to-date global data.
Highlights:
• Migrated the client’s on-prem data management to a cloud-based Azure system, enabling near real-time analytics and reporting
• Reduced processing time of nearly one million rows by 93%
•
Minimized technical skills required to use the tool
•
Reduced overall cost by 45%, saving the client between $5-6K a month