Tuesday, June 30, 2020

Reduce Project Time and Complexity with DataOps

Business Case:

Operational excellence in IT requires regular monitoring of multiple tasks. Effectively tracking tasks is particularly important when projects handle and process real-time data. 

A typical DataOps environment might require tracking job statuses, database health, database access, server health, CPU usage, data pipelines, issues, and much more.

Monitoring becomes complicated when multiple development teams work under the same project, handling different streams and phases. Cross-team data, resource, and deployment interdependencies require a high degree of collaboration. Factors like refresh frequencies, requisite SLAs, and shifting stakeholder priorities further complicate project structures. The effort required to manually track and monitor so many details is time consuming, inefficient, and costly.

Key Challenges:

  Simplify the monitoring and tracking interface 
  Reduce manual intervention points 
  Reduce process complexity 
  Improve collaboration among dependent teams 
  Fix recurrent issues and streamline data flow 

Our Solution:

We used Power BI and Power Apps to create a single dashboard that includes all project parameters and status updates.

Figure 1: Solution Design
 We used Microsoft Power BI reports to track real-time data from SQL Server applications, including:

  Job running status for on-premises and cloud services (completed, in progress, failed, scheduled, disabled) 
  Job history 
  Database status (health, stale tables, job trends, data latency, database access) 
  Server status (health, memory usage, CPU usage) 

We created a Power Apps based application to register and track any issues and their status (open vs. resolved). Our solution lets users enter the cause of an issue, and the action and effort required to resolve it.

Business Outcomes:

A single-point view reduced the time and complexity of tracking and monitoring 40+ streams, 250+ jobs, 150+ databases, and 70 servers in real time. The interface facilitated collaboration across all dependent teams on a project. 

Our solution enables us to retrospect, analyze, identify, and isolate the root cause of recurrent issues and process bottlenecks. We can now fix such issues with less manual effort. By limiting manual touchpoints, we improved our efficiency by over 60%. Our solution also optimized resource utilization, lowered chances of error, and improved quality control.


   The single-point view stream reduced the time and complexity of tracking and monitoring jobs, databases, and servers
    Our unified interface improved overall efficiency by over 60%, with optimized resource utilization, lower error rates, and better quality control