Friday, November 4, 2022

ADX Implementation for a global Cloud Management Software provider

Business Case

Our customer, a global Cloud Management Software provider, helps IT service providers to build successful cloud businesses. The customer’s primary goal was to make its partners security compliant. Customer’s partners measured the compliance status by means of three different security metrics namely Current Average Security Score, Average Security Score, and Max Recorded Security Score.

Our customer used to receive numerous requests from its partners in JSON file format to process the security scores. The existing system was not scalable to respond to all requests simultaneously. To resolve the problem, the customer wanted a solution that could handle multiple requests and refresh data for near real-time reports.

Key Challenges

  Optimize and scale existing data pipelines to handle up to 250 requests simultaneously 
  Near real-time data refresh for batch requests 
  Publish real-time reporting insights for further analysis and decision-making 

Our Solution

Deep dive analysis was performed to identify solutions to overcome the key challenges.

For pulling data from upstream sources and sending data to storage, we carried out explorations using Azure Data Factory (ADF), Logic Apps, and Function Apps. We finalized ADF as the most suitable option amongst the three options.

For data storage and processing, we identified that Azure Data Explorer (ADX) provided the required degree of real-time reporting in contrast to other options. We designed a cost-efficient solution using ADX that was reliable and secure.

To ensure that there are no performance issues with the system while maintaining a queue of requests, we used Azure Event Grids. The pipelines were implemented with a capability to handle both scheduled batch loads and just-in-time requests. With scheduled batch loads, data was pulled for a defined timeframe and for just-in-time (JIT) requests, data was pulled once the request was received.

To secure data transmission, secret key combinations were used to send calls to the Event Grid. All transmissions to and from Power BI Service were encrypted.

Power BI Embedded was used to publish reports to the web application. Row Level Security (RLS) was implemented for enhanced security.

Key Technologies

Azure Functions, Azure Data Explorer, Azure Data Factory, Azure SQL Database, Azure Monitor, Azure Front Door/Traffic Manager, Power BI Embedded, Event Grid, Storage Accounts, App Service. 

Business Outcomes

  Provided near real-time analytics with minimal cost for the customer as compared to initial system. 
  Scalable design and architecture that led to adoption of similar design for the remaining products of the customer. 
  Cost incurred by the customer for implementation of the capability was very low.