Friday, November 6, 2020

Quickly Close Business Gaps with Near Real Time (NRT) Analytics

Business Case:

Our client, a multinational software company, relies on data analytics to identify performance gaps and improve business operations. Prior to our solution, our client processed data using batch execution, which runs continuously in a sequential fashion. Due to our client’s large volume of data, report execution often exceeded 3 hours and delayed critical business decisions. In partnering with us, our client’s goal was to reduce refresh time to 30 minutes, enabling their business users to access data in near real time (NRT).

Key Challenges:

  Reduce data latency  
  Ensure reports are updated within 30 minutes  

Our Solution:

We used Azure Stream Analytics (ASA) and Azure Databricks (ADB) to reduce data latency by approximately 84%. 

Figure 1: Solution Design

Our solution was implemented using lambda architecture, designed to process massive volumes of data through hybrid batch-processing and stream-processing methods. The Spark SQL engine runs the architecture incrementally, continuously updating the data. 

We source data from the Azure Event Hub, where refreshes originate. Refreshes are captured through stream analytics and the updated data is pushed to Azure Data Lake Storage (ADLS). The data is processed in ADLS, then pushed to Power BI.  

Business Outcomes:

Implementing a hybrid data processing method using ASA and ADB reduced our client’s data latency by approximately 84%.  

Real-time data analysis dramatically increases business leaders’ ability to predict outcomes and strategize alternatives. Imagine, a manager wants to know if they are the assigned lead on a project. With a three-hour data delay, the manager may miss important project deadlines. Now, with our NRT solution, the manager can access the information in under 30 minutes. Real-time reporting enables our client to effectively allocate resources and respond to growing business needs.

Highlights:

    Implemented a hybrid data processing method using Azure Stream Analytics (ASA) and Azure Databricks (ADB)
    Reduced data latency by 84%
    Ensured reports update within 30 minutes