November 18, 2024

Transforming data operations with Microsoft Fabric and CI/CD for ISVs

Share

   












In today’s fast-paced tech landscape, data drives decision-making and innovation. Migrating organizational data to Microsoft Fabric is a transformative move. Microsoft Fabric provides a robust platform that integrates seamlessly with existing data sources, creating a unified, scalable data environment. Using Microsoft Fabric brings multiple advantages, including unified Delta data format, data democratization, integrated AI (artificial intelligence), and simplified security management.

 


About CI/CD

Building software goes beyond writing code. Continuous Integration and Continuous Deployment (or Delivery)—known as CI/CD—forms essential practices for modern tech stacks, providing a superior alternative to manual deployments. CI/CD enables a streamlined process, delivering faster, cleaner, and more reliable software releases.

Advantages of CI/CD

·        Accelerates release cycles and time-to-market: CI/CD automates integration, testing, and deployment, enabling rapid response to customer needs.

·        Ensures a consistent and reliable process: Automated testing and deployment improve software stability and consistency, reducing errors.

·        Scales to manage environments effectively: CI/CD simplifies managing multiple environments, allowing seamless scaling as projects grow.

·        Boosts developer productivity: By reducing manual tasks, CI/CD lets developers focus on coding, leading to faster development and more efficient teams.



How Microsoft Fabric enables CI/CD

·        Built-in Git integration: Sync Microsoft Fabric workspaces with a Git branch (Azure DevOps or Git) for streamlined version control and collaboration.

·        Microsoft Fabric deployment pipelines: Out-of-the-box deployment pipelines support 2 to 10 stages, enabling flexible deployment across multiple workspaces. Sync code forward or backward across stages to ensure consistency.

·        Microsoft Fabric REST APIs: Automate tasks using REST APIs (application programming interface) that support user accounts, managed identities, or service principals (SPNs) for seamless, secure operations.



Implementation for ISVs

Promoting solutions and managing deployments for ISVs

ISVs (independent software vendors) have a few options to promote their solutions to customers:

·        Deploying multi-tenant applications within the ISV tenant for customer use.

·        Publishing Microsoft Fabric Apps.

·        Publishing Microsoft Fabric workloads.


In this case, an ISV deployed a multi-tenant application using Microsoft Fabric’s Platform as a Service (PaaS) capabilities, delivering Software as a Service (SaaS) to customers. Microsoft Fabric allows the ISV to provide reliable software, with CI/CD enabling smooth updates across hundreds of customer workspaces. Beyond version control, automated testing, and Infrastructure as Code (IaC), ISVs require:

·        Multiple environments and configurations.

·        Scalable pipelines.

·        Robust monitoring and observability.

·        Granular control over artifacts deployed to each customer workspace.


Microsoft Fabric REST APIs addressed these needs, supporting streamlined, controlled deployments.



Infrastructure as code

Treating workspace items as building blocks is essential for solution implementation. Each customer workspace is configured based on data profiles, optimizing resource use. A central Lakehouse stores these configurations, and resources are provisioned accordingly. Workspace provisioning uses a template and configuration table stored centrally in the Lakehouse. The infra team provisions workspaces using REST API calls, creating custom pools for customers and deploying environment objects as needed.



Code deployments

Figure 1: Architecture

Infrastructure as code

The deployment approach is similar to trunk-based branching strategies. For ISVs where granular item control is required, Microsoft Fabric REST APIs are the best way to manage deployments. The deployment process includes:


1.     Merging pull requests from feature branches to the main branch.

2.     Building, testing, and triggering a release pipeline to deploy to the Dev workspace. After deployment, a post-deployment call updates item configurations in Dev.

3.     Repeating the process in QA (quality assurance) workspaces to simulate production.

4.     Deploying to production workspaces after testing. Each customer has a unique workspace, allowing parallel deployments. Post-deployment API calls handle resource mapping for artifacts.


Notes:

1.     Lakehouses are created blank, with tables and shortcuts generated automatically via a notebook execution API.

2.     Parameterization, like Azure Data Factory, is currently unavailable in Microsoft Fabric. Connections are resolved through autobinding if Lakehouses are provisioned before pipeline deployment.

3.     Connections are currently not Microsoft Fabric objects, requiring manual setup during deployment.



What’s next

While CI/CD processes are implemented as outlined, further improvements could streamline it:

1.      Configuration variables as Microsoft Fabric items: Adding configuration values as Microsoft Fabric items would simplify and standardize settings.

2.      Azure Data Factory-style templatization: Managing connections as Microsoft Fabric items would improve flexibility.

3.      SPN support for all APIs: Adding SPN authentication across all APIs would align with production best practices.

4.      Support for additional Git providers: Expanding to GitLab and Bitbucket would increase Microsoft Fabric’s adoption by aligning with existing workflows.



Interested in learning more?

As a Microsoft Fabric Featured Partner, MAQ Software brings deep expertise in helping organizations unlock the full potential of Microsoft Fabric. Whether you're looking for guidance on implementing data solutions or optimizing your existing platform, we’re here to support you every step of the way. Contact us at Sales@MAQSoftware.com or explore our apps or consulting services on Microsoft Azure Marketplace: