How to Use Azure Services: Product Comparison

With over 50 Azure services out there, deciding which service is right for your project can be challenging. Weighing the pros and cons of each option for numerous business requirements is a recipe for decision paralysis. When it comes to optimizing your Azure architecture, picking the right tool for the job is key.

Over the last decade, MAQ Software has migrated hundreds of clients to Azure. Along the way, we’ve picked up Azure tips and tricks that enable us to develop systems that are faster, more reliable, and operate at a lower cost. In this article, we’ll explore the differences between Azure services so you can pick the one that’s right for you.

Which Azure cloud storage service should I use?

Azure Data Lake Storage (ADLS) Gen 1 vs. ADLS Gen 2 vs. Blob Storage

When to use Blob storage instead of ADLS Gen 2: When your project involves minimal transactions, use Blob storage to minimize infrastructure costs. For example: production backups.

When to use ADLS Gen 2 instead of Blob storage: When your project has multiple developers or involves analytics, and you need to securely share data with external users, use ADLS to leverage Azure Active Directory authentication. This prevents unauthorized users from accessing sensitive data. For example: global sales dashboards.

When to use ADLS Gen 1 instead of ADLS Gen 2: When your project executes U-SQL within Azure Data Lake Analytics (ADLA) on top of storage services, use ADLS Gen 1. This is useful for projects that need analytics and storage performed from a single platform. For example: low-budget implementations.

When to use geo-replication: Geo-replication is available on all Azure cloud storage services. As a rule of thumb, avoid geo-replicating the development environment to keep infrastructure costs down. Only implement geo-replication for production.

Which Blob storage access tier to use: Picking the optimal access tier is important to achieve your desired performance at minimal storage costs:

Tier	Key differentiator	When to use it
Hot	Optimized for frequent access to objects in the storage account	For projects that require daily refresh and frequent data transactions Example: cloud data with daily refresh
Cool	Optimized for storing large volumes of data that is infrequently accessed and stored for at least 30 days	For projects that require monthly fresh with limited transactions Example: monthly snapshots
Archive	Optimized for storing large volumes of data that is infrequently accessed and stored for at least 180 days	For projects that require static data, snapshots, and/or yearly refresh storage with almost no transactions Example: yearly snapshots

Table 1: Blob Storage Access Tier Comparison

Which Azure cloud processing service should I use?

Azure Databricks vs. Azure Synapse Analytics

When to use Azure Databricks: When your project has multiple developers or involves analytics, and you need to securely share data with external users, use ADLS to leverage Azure Active Directory authentication. This prevents unauthorized users from accessing sensitive data. For example: global sales dashboards.

•    Deep learning models: Azure Databricks reduces ML execution time by optimizing code and using some of the most popular libraries (e.g., TensorFlow, PyTorch, Keras) and GPU-enabled clusters.

•    Real-time transformations: Databricks Runtime supports Spark's structured streaming and autoloader functionality, meaning it can process stream data such as Twitter feeds.

When to use Azure Synapse Analytics: When your project uses SQL analyses and data warehousing, or reporting and self-service BI. If you need to process large volumes of data without an ML model, use Azure Synapse Analytics:

•    SQL analyses and data warehousing: Synapse is a developer-friendly environment, supporting full relational data models and stored procedures, and providing full standard T-SQL features. When migrating data from on-premises to the cloud, use Synapse for a seamless transition.

•    Reporting and self-service BI: Power BI is integrated directly into the Synapse studio, which reduces data latency.

Which Azure Databricks pricing tier to pick: Choosing the right pricing tier is important to reduce infrastructure costs while achieving the desired performance:

Pricing Tier	Key differentiator	When to use it
Premium	The premium tier supports role-based access to notebooks, clusters, and jobs	For projects that involve mulitple stakeholders in a shared development environment For example: one platform accessed by multiple vendors
Standard	The standard tier costs ~20% less than the premium tier	For projects that involve a single stakeholder and limited development teams Example: one platform run and developed by a single vendor

 Table 2: Databricks Pricing Tier Comparison

Which Azure Databricks workload to use: Choosing the right workload type is important to ensure you achieve the desired performance at a reduced cost:

Workload	Key differentiator	When to use it
Data Analytics	Data Analytics is more flexible as it supports interactive clusters and an analytics ecosystem	Suited for dev environments, which require higher collaboration and multi-user sharing
Data Engineering	Data Engineering costs ~30% less than Data Analytics	Suited for UAT/prod environments, which generally require less collaboration and can be run by a single developer

 Table 3: Databricks Workload Comparison

How to choose Databricks cluster size: For development purposes, start with a smaller Databricks cluster (i.e., general purpose) and enable auto scaling to optimize costs; based on specific needs, you can opt for higher clusters.

Which Azure data expose service should I use?

Azure SQL database vs. Azure Synapse (formerly Azure Data Warehouse)

When to use Azure Synapse: When projects deal with large volumes of data (>1 terabyte) and small number of users, or use OLAP data. For example: global sales data.

When to use Azure SQL database: When projects work with real-time data (max 1 terabyte) and many users, or use OLTP data (application database). For example: ATM transaction.


Still have questions? Feel free to reach out: Support@MAQSoftware.com

Do you want to know more about our data management best practices, projects, and initiatives? Visit our Data Management Hub to learn more.

Resolve Support Tickets Faster with Predictive Risk Algorithms

Business Case:

Our client, a global technology company, receives approximately 15 million customer support tickets a year. With an overwhelming volume, it was impossible for support engineers to identify which tickets were at highest risk of breaching service-level agreement (SLA). As a result, high-risk tickets could go unresolved for weeks, creating a major backlog.

At the time of partnering with us, our client had approximately 10 million open support cases in their ecosystem. Our client needed a solution that enabled support engineers to resolve tickets faster.

Key Challenges:

•  Automatically assign ticket priority  

•  Accurately predict high-risk cases  

Our Solution:

We implemented a CatBoost algorithm that uses machine learning to prioritize tickets based on risk probability.

Using historical ticket data, we identified the categorical and numerical attributes that our algorithm would evaluate to determine risk probability. The identified attributes included ticket type, ticket severity, initial response time, total time in queue, SLA state, and customer satisfaction score.
After comparing numerous algorithms, we concluded that CatBoost (algorithm for gradient boosting on decision trees) provided the most accurate results. Compared to other models, CatBoost:

•  Allows the most categorical attributes  

•  Saves time with faster model loading (does not require time-consuming one-hot encoding and pre-processing)  

•  Is available at a lower cost  

•  Requires a minimal learning curve  

To launch the model, our team used data outliers to inform the algorithm. With CatBoost’s machine learning capability, the algorithm increases its accuracy over time.

Figure 2: Mobile Notification

To improve support engineers’ response time, we implemented an integrated web feature that sends high-risk ticket notifications to engineers’ cell phones.

Business Outcomes:

Our CatBoost algorithm implementation enabled support engineers to resolve high-priority tickets faster. The algorithm uses machine learning to accurately assign ticket priority based on comprehensive risk evaluation. Prior to our solution implementation, our client had a backlog of 10 million tickets. Now, support engineers can reduce their backlog while keeping up with incoming tickets. With our added cell phone notifications, support engineers can immediately respond to high-risk tickets and ensure SLA compliance. Overall, our solution has enabled our client to significantly improve customer satisfaction.

Highlights:

•    Implemented a CatBoost algorithm that automatically prioritizes tickets based on risk probability

•    Reduced ticket resolution time

•    Added a cell phone notification for high-risk tickets

For more on customer support, check out how our custom Dynamics 365 portal improved our client's gaming support operations

Create a Single Source of Truth with Dynamics 365

Business Case:

Our client leases data centers across the globe. To maintain their leases, our client coordinates with up to 100 lease providers, which involves multiple methods of communication. In addition, a single lease can require as many as 20-30 documents.

Previously, our client tracked center details across multiple different spreadsheets. Quotations, site progress, and updates were managed over email and through notations from in-person visits. Our client struggled to consolidate data and track updates. Our client needed a centralized, easy-to-maintain solution that provided one source of truth for global information.

Key Challenges:

•  Consolidate assets into a single location, including emails, documents, and meeting notes 

•  Integrate existing SharePoint system with the solution  

•  Enable users from different regions to easily access global data center details such as sites, lease providers, and geographies  

Our Solution:

We created a Dynamics 365 portal that consolidates and presents details from email, Excel spreadsheets, OneNote, and SharePoint Online.  

Figure 1: Solution Design

Our solution uses Dynamics 365 App for Outlook to link to our client’s primary mail system. Users can track Outlook emails and appointments directly from the Dynamics 365 environment. The portal consolidates all global data center information, which can be accessed from different geographies.  

During our Outlook implementation, we identified an opportunity to enable our client to manage data directly in Dynamics 365. Understanding the value in this capability, our client expanded the solution scope. We added entities (tables in the Dynamics 365 environment) to replace our client’s existing Excel records. Dynamic 365’s built-in auditing capabilities allow our client to track edits in real time. Users can also import Excel spreadsheets directly into the portal rather than manually inputting information. We added an extension that enables users to upload OneNote and SharePoint Online files to the Dynamics 365 portal.  

We implemented role-based security to protect our client’s sensitive data. There are three main roles: 

1.  Admin: Can access, update, create, and delete all records. 

2.  User: Can access, update, create, and delete any records within their specific territory/project. 

3.  Reader: Can view all records, but does not have editing permissions. 

Our solution uses the Dynamics 365 export service to auto-synchronize our client’s SQL database. The SQL database acts as an import model for Power BI reports that generate key leadership insights. To ensure data is validated securely from end to end, we implemented Power Automate flows. If the uploaded data contains errors, our solution sends an email notification to managers, who can take immediate corrective action.  

Business Outcomes:

Our centralized Dynamics 365 platform enabled our client to track appointments, emails, documents, and notes across the entire team. By developing a single source of truth, our client avoids miscommunication and rework, reducing data management efforts by 60%. Maintaining data sets is easier, as all information can be accessed from one location. With role-based security and edit tracking, our client can monitor access and update permissions.  

We integrated data from multiple sources, which enables field workers operate seamlessly. For example, a field worker might deal with a data center that operates out of Paris. All team members can view emails and appointments from our centralized platform. If the field worker then visits the Paris data center and records meeting notes on OneNote, these can also be uploaded directly to the Dynamics 365 portal. All relevant data will be automatically associated with the Paris data center records. 

Our solution reduces data management and tracking efforts by 4-6 hours a week. Since implementation, our portal has supported multi-channel communication between our client’s 50 team members and the nearly 100 data center providers around the world.

Highlights:

•    Developed a Dynamics 365 portal that centralizes data across emails, Excel spreadsheets, OneNote, and SharePoint Online

•    Eliminated manual tracking, reducing data management efforts by 60%

•    Enabled seamless communication between 50 team members across the globe, reducing efforts by 4-6 hours a week

•    Simplified data maintenance by enabling users to edit data directly in the portal

•    Implemented role-based security

Read More: Track Dynamics 365 Implementation Updates More Efficiently with Data Dictionary

Engineer Product Features in Real Time Based on Customer Feedback

Business Case:

In today’s consumer-centric environment, the best way to improve a product is by responding to customer feedback. A typical customer feedback cycle requires cross-team communication between support, sales, and engineering. The sales team offers insights into product features that receive recurring feedback; the engineering team has the technical expertise to implement changes and offer support. Responding quickly to feedback is critical to ensure customer satisfaction.

Our client, a global technology giant, stores support ticket information in Azure DevOps. Our client’s sales team receives thousands of support cases a day, but only converted 600 cases a year into engineering features. Without an effective classification and communication system, feedback went unresolved. Although the sales team was logging customer feedback, they couldn’t connect with the engineering team, who often didn’t know the feedback existed. In partnering with us, our client needed to bridge the gap between their sales team and engineering team.

Key Challenges:

•  Define a feedback triage process for the engineering team 

•  Effectively share feedback between teams  

•  Create a reporting system where engineers can access all relevant feedback data  

•  Reduce feedback response time 

Our Solution:

We developed a two-part solution to connect our client’s sales and engineering teams: 

1.  Establish a feedback triage process 

2.  Create a Power BI dashboard that pulls feedback information from the engineering DevOps 

Figure 1: Solution Design

The triage process begins by pulling customer feedback from two different sales portals into a single DevOps workspace. During level one triage, the engineering team verifies the feedback Work Item is complete with pre-defined metrics such as impact, customer scenario, and description. If the Item lacks key information, the team sends it back to the requestor.  

As part of level two triage, the engineering team meets weekly to assess feedback Items. If an Item needs additional information, the team sends it back to the requestor. Otherwise, the team tags the feedback as “Evidence” and attaches it to an engineering Feature. Engineers respond to the Features directly through DevOps. One Feature might contain dozens of Evidence Items based on repeat customer feedback. To ensure Features aren’t duplicated, we created a separate table in an Azure Databricks notebook that maps new feedback with existing Features.  

Figure 2: Engineering Feedback Dashboard Home Page (using sample data)

With a triage process in place, we then developed a dashboard where engineers could view and prioritize all open Features. The dashboard pulls data from DevOps using API calls in Azure Databricks. Our solution uses custom calculations to derive key metrics from the data. Once transformed, data is pushed to Azure SQL tables that populate the Power BI dashboard. The dashboard includes information such as revenue impact and customer profile so engineers can effectively prioritize feedback response. Engineers can group data by sector, product type, country of origin, and many other metrics using the dashboard’s multiple tabs.  

Figure 3: Feedback by Revenue Impact Drilldown (using sample data)

Business Outcomes:

Our engineering triage process and Power BI feedback dashboard enable our client to close the gap between sales and engineering. With an effective triage process, the engineering team can quickly access relevant data. To increase response speed, we group repeat queries under a single Feature. With numerous impact views, the dashboard enables engineers to effectively prioritize tasks. Customers receive faster responses to high-priority feedback.  

Increased accountability improves the team’s ability to collaborate on technically complex feedback. Our client can continually enhance products and services based on customer requirements. Effective feedback response enables our client to build long-lasting customer relationships and identify potential revenue opportunities.  

Since implementation, our client has created over 1K feedback Items, increasing DevOps usage by 66% in under five months. Over 300 of these items have been converted into Features for the engineering team to work on. The dashboard is used across the engineering team, with over 1K views each month.

Highlights:

•    Developed an engineering triage process and Power BI dashboard that enable the engineering team to respond to customer feedback

•    Increased use of the engineering DevOps workspace by 66%

•    Enabled engineers to prioritize customer feedback with a dashboard that has over 1K monthly views

•    Reduced feedback response time

Read More: Connect High-Impact Service Opportunities with Underutilized Support Team Resources

Manage and Share Events with a Custom Calendar App

Business Case:

Our client, a global technology company, organizes thousands of marketing events each year. To launch an event, marketing managers need to add the event to the calendar, record details, and present it to the planning team using PowerPoint. Due to our client’s massive volume of data and various capability requirements, managers needed to use separate platforms to complete their tasks. With multiple data sources, managers were unable to completely view and filter events to ensure effective execution.

Unfortunately, no single platform currently on the market provides the capabilities and data capacity our client requires. In partnering with us, our client’s goal was to design a centralized calendar solution that enables managers to access and export event information from a single location.

Easily Embed Power BI Content into ReactJS Apps with an Open Source Wrapper

Microsoft’s Power BI is a premier data visualization tool. Most of our Fortune 500 customers use Power BI for enhanced reporting and analytics. Power BI Embedded is an Azure Service that enables developers to embed Power BI content (dashboards, reports, tiles, and visuals) into an application.

ReactJS is the most popular frontend framework for building user interfaces (UIs). Within the ReactJS community, there is high demand to easily embed Power BI content. Prior to this solution, users needed to maintain both the ReactJS app and Power BI embedding lifecycles, leading to time-consuming rewriting of code. Driven to adopt and innovate the latest technologies, the Microsoft Power BI Embedded product team partnered with us to create an open source solution.

Solution

We assisted Microsoft’s Power BI Embedded product team to develop an open source wrapper that enables users to easily embed Power BI content into React-based applications. The wrapper consists of a component library called powerbi-client-react, which eliminates the need for writing large amounts of code.

The wrapper was implemented in TypeScript and is compiled to JavaScript. You can consume the wrapper from NPM or find the source code on GitHub.

Figure 1: Solution Design

Use Cases

•    Embed Power BI content into a React-based application

•    Reference embedded content

•    Apply cascading style sheets (CSS) class style

•    Set and reset event handlers

•    Set new access token after expiry

•    Update embed settings

•    Bootstrap Power BI content

•    Author Power BI reports

•    Phased embedding

To see the wrapper in action, check out the DEMO on GitHub.

Benefits

•    Reduces development effort by approximately 20%

•    Lightweight: 39 kB (unpacked)

•    Supports embedding of all Power BI content

•    Supports all features of Power BI Embedded

•    Provides smart props that manage the complete embedding lifecycle

•    Wrapper components can be used as independent plug and play components

•    Compatible with Power BI report authoring library, enabling an application to edit the report from the embedded report

•    Hosted on NPM and open sourced on GitHub under MIT License

At the time of publishing this article, the powerbi-client-react wrapper has 2,216 weekly downloads on NPM, and has 77 “Stars” and 12 “Forks” on GitHub.

Quickly Close Business Gaps with Near Real Time (NRT) Analytics

Business Case:

Our client, a multinational software company, relies on data analytics to identify performance gaps and improve business operations. Prior to our solution, our client processed data using batch execution, which runs continuously in a sequential fashion. Due to our client’s large volume of data, report execution often exceeded 3 hours and delayed critical business decisions. In partnering with us, our client’s goal was to reduce refresh time to 30 minutes, enabling their business users to access data in near real time (NRT).

Key Challenges:

•  Reduce data latency  

•  Ensure reports are updated within 30 minutes  

Our Solution:

We used Azure Stream Analytics (ASA) and Azure Databricks (ADB) to reduce data latency by approximately 84%. 

Figure 1: Solution Design

Our solution was implemented using lambda architecture, designed to process massive volumes of data through hybrid batch-processing and stream-processing methods. The Spark SQL engine runs the architecture incrementally, continuously updating the data. 

We source data from the Azure Event Hub, where refreshes originate. Refreshes are captured through stream analytics and the updated data is pushed to Azure Data Lake Storage (ADLS). The data is processed in ADLS, then pushed to Power BI.  

Business Outcomes:

Implementing a hybrid data processing method using ASA and ADB reduced our client’s data latency by approximately 84%.  

Real-time data analysis dramatically increases business leaders’ ability to predict outcomes and strategize alternatives. Imagine, a manager wants to know if they are the assigned lead on a project. With a three-hour data delay, the manager may miss important project deadlines. Now, with our NRT solution, the manager can access the information in under 30 minutes. Real-time reporting enables our client to effectively allocate resources and respond to growing business needs.

Highlights:

•    Implemented a hybrid data processing method using Azure Stream Analytics (ASA) and Azure Databricks (ADB)

•    Reduced data latency by 84%

•    Ensured reports update within 30 minutes