Increasing efficiency by switching from batch processing to real-time streaming

 

About our client

Established in 1978, our client is an industrial technology company. With 2000+ patents for technological innovations, they provide integrated technology and software to the world’s largest industries. From purpose-built products to enterprise lifecycle solutions, they are transforming industries such as agriculture, construction, geospatial, and transportation.

The issue at hand

In their construction department, their comprehensive solutions span the entire construction lifecycle, from planning and design to construction and maintenance. These solutions enable their customers to have seamless data integration and real-time collaboration. However, the construction department faced several business challenges:

•  Inefficient estimating processes led to inaccurate bids and cost overruns.

• Fragmented project management resulted in poor collaboration and inadequate risk management.

• Suboptimal financial management caused disconnected job costs and burdensome manual processing.

The client’s initial solution and how we stepped in

Initially, they transitioned to a cloud-based ERP solution using Databricks and Snowflake. This resolved many issues by providing quicker, more accurate estimates and improving bid profitability with integrated digital takeoff and cost databases. However, the setup had significant drawbacks too. The primary issue was the lack of real-time data updates, hindering timely decision-making and operational efficiency. Additionally, the solution required complex integration and maintenance, leading to increased operational overhead and slower response times. The fragmented system also resulted in data silos, complicating effective collaboration and risk management.

These limitations highlighted the need for a more cohesive and efficient solution. This solution needed to provide true real-time intelligence and seamless integration with their existing setup. We suggested a solution with Microsoft Fabric's Real-Time Intelligence (RTI) capabilities which emerged as the ideal choice due to its comprehensive, all-in-one nature. It eliminated the need for new components and used the company's existing Power BI integration. This made the implementation of architecture and security swift and straightforward.

With Fabric RTI, the company now benefits from real-time data updates. This enables more efficient decision-making, and improved collaboration as well as operational performance.

 

Diving into the new Fabric setup

In our project, we implemented a comprehensive real-time intelligence solution using Microsoft Fabric's capabilities. This setup enabled seamless data ingestion, processing, and analysis, providing immediate insights and automated responses to data anomalies. Here are the detailed steps of our implementation:

Eventstream - Ingestion and processing

·    We used Eventstreams within Microsoft Fabric RTI to ingest data from Azure SQL database into Fabric.

·    The ERP system data stored in Azure SQL database was captured using the Azure SQL Database CDC source connector. This provided snapshots of the current data.

·    Kafka was employed to stream this ERP data into the system.

·    Eventstreams efficiently converted data from JSON into a table. This addressed Kafka metadata issues to ensure seamless data integration.

 

Eventhouse - Storage and processing

·    Once ingested, the data was transferred to Eventhouse, which managed high-velocity data streams.

·    Eventhouse ensured secure and efficient access and processing of the stored data. This allowed a high level of performance and reliability to be maintained.

·    This setup managed large volumes of streaming data, supporting our real-time analytics requirements.

 

KQL Queryset - Data management and transformation

·    KQL Queryset managed the data by applying policies for transformations. This ensured data integrity and consistency was maintained.

·    We used scheduled pipelines within KQL Queryset to regularly clean up tables. This allowed data to remain accurate and up-to-date.

·    KQL was used to process and transform data stored in Eventhouse. With it, easy query execution and results customization was enabled.

 

RT Dashboard - Real-time analytics and visualization

·    For real-time analytics and visualization, the RT Dashboard using Power BI was used. This is a component of the Real-Time Intelligence feature in Fabric.

·    The dashboard allowed us to monitor, visualize, and analyze data in real time. Immediate insights were provided through various widgets and visualization tools.

·    We built reports on the RT Dashboard to monitor ERP metrics such as inventory levels, order fulfillment, cash flow, and supplier performance in real time.

 

Reflex - Monitoring and action

·    Reflex continuously monitored the data streams for defined conditions. It triggered corresponding actions immediately upon detecting anomalies.

·    This setup ensured timely alerts and resolutions for any issues. It maintained the smooth operation of the system and prevented potential disruptions.

·    Reflex played a crucial role in automating responses to data anomalies. The overall efficiency and reliability of the real-time intelligence solution was improved.

 

OneLake and Real-Time Hub - Centralized data management and access

·    OneLake served as a central repository for the ingested and processed data, providing a unified storage solution.

·    The Real-Time Hub facilitated seamless integration and interaction between various components of the system. It ensured efficient data flow and access. 

 

Figure 1: Components of Fabric’s Real-Time Intelligence

With this implementation approach, we successfully used Fabric's capabilities to create a robust and efficient real-time intelligence solution. This setup streamlined the architecture and security implementation by using existing storage and Power BI infrastructure to improve the solution’s efficiency and effectiveness.

Streamlined data ecosystem delivers real business value

Our client achieved significant business improvements through our real-time intelligence solution using Microsoft Fabric. By consolidating data pipelines from tools like Databricks and Snowflake into a unified Fabric platform, we streamlined their data ecosystem and delivered following results:

·    Cost savings: Streamlined data ecosystem, resulting in 20-30% cost savings.

·    Simplified maintenance: Reduced IT maintenance efforts by approximately 50%.

·    Improved data processing speed: Reduced latency from 60 seconds to 6 seconds, enabling faster reactions to market shifts.

·    Better decision-making: Real-time dashboards with Power BI provided immediate insights, transforming decision-making across the organization.

·    Proactive issue resolution: Reflex monitored data streams and triggered alerts for anomalies, allowing proactive issue resolution.

Harnessing the power of real-time intelligence in Fabric

In today's fast-paced business environment, real-time insights are crucial for staying ahead of the curve. Microsoft Fabric’s based real-time Intelligence solution empowers organizations to unlock this potential by:

·    Seamlessly ingesting data from various sources.

·    Streamlining data pipelines and reducing costs.

·    Reacting to changes and opportunities much faster with near real-time processing capabilities.

·    Fostering data-driven decision-making with real-time dashboards.

·    Ensuring top-notch security with industry-leading encryption standards and granular access controls.

By providing all these features, Microsoft Fabric Real-Time Intelligence helps businesses gain a competitive edge in the era of real-time data.

For any further inquiries, contact Sales@MAQSoftware.com to see how a real-time intelligence solution powered by Microsoft Fabric can transform your business, improve customer satisfaction, and unlock significant cost savings.