June 16, 2022

Enhancing fintech analytics to provide millions of borrowers with better loan options

Share

Business Case

Our client, a leading fintech company, enables thousands of financial institutions to engage millions of borrowers with better loan options. Our client was on a mission to expand their analytics platform when they faced a critical block: Their existing platform architecture was at maximum data capacity. To onboard new customers, our client needed a more scalable analytics solution. In addition, our client wanted to enhance their platform’s reporting experience. Existing reporting was limited and required users to export data to Excel for manual analysis, delaying insights. To increase their product value and onboard more customers, our client needed a scalable architecture with embedded reporting.


Key Challenges

•  Enable analytics platform to scale to 1000+ customers 
•  Enable self-serve, near real-time analytics 
•  Enable AI/ML capabilities for future innovation 
•  Improve the security of financial data 


Our Solution

We rebuilt our client’s analytics platform using Azure Synapse, Azure Data Lake Storage, Azure Data Factory, Azure Databricks, and Power BI. To ensure operational and technical excellence throughout the build, we followed the five pillars of the Azure Well-Architected Framework and leveraged migration strategies from Microsoft’s Cloud Adoption Framework.

Reliability: Implemented query replica within Azure Analysis Services (AAS) to ensure resource-intensive queries do not impact ETL processing. Configured secondary and backup resources to ensure 100% resource availability.

Security: Enabled role-based access and disabled public access to storage accounts with PII data to ensure partner data is isolated within the ecosystem. In doing this, we greatly reduced the risk of security threats.
Cost Optimization: Implemented auto-scaling in lower environments, enabled Databricks to scale down when inactive, and deployed Power BI report on cost monitoring to scale services as needed.

Operational Excellence: Created Terraform automated scripts for Azure resources deployment. Implemented proactive monitoring for pipeline bottlenecks, ETL execution, and failures.

Performance Efficiency: Implemented parallel processing and concurrent querying of underlying data model for 1000+ customers using Azure Databricks.

In addition, our automated deployment framework uses continuous integration / continuous delivery (CI/CD) pipelines to create Azure landing zones by focusing on identity, network, and resource management. To deploy Azure landing zones, we used a proprietary approach that combines the benefits of both the “start small and expand” and the “enterprise-scale”. Using industry-standard best practices and our center of excellence for Azure infrastructure setup, we ensured the right configuration to build a strong foundation and avoid rework in the future. This approach reassures our customers about our capabilities while creating a secure and reliable environment that is built to last.


Governance and Compliance

Governance in analytics, especially in the financial sector, is pivotal. It’s critical for data protection, ensuring regulatory compliance, managing risks, and maintaining stakeholder trust. Here's how we prioritized governance in our solution:

Row-level Security (RLS): Understanding the sensitivity of financial data, we employed RLS in Power BI, as mentioned previously. This ensured that data access was granular, with users only seeing data relevant to their roles. RLS protects customer and financial information at a micro-level.

Data governance frameworks: Using Power BI's capabilities, we implemented a structured data governance framework. This framework classified and labeled data, ensuring only authorized personnel had access to specific data sets. Certified datasets were promoted, guiding users to trusted sources of data and reducing the risk of inaccurate analysis.

Audit and monitoring: With integration to Microsoft 365's compliance center, all Power BI activities are monitored, creating an audit trail. This not only facilitates regulatory compliance but also helps in detecting and addressing any irregularities.

Policy adherence: With the client's internal compliance teams, we established and enforced data usage policies. Using Azure Purview, we efficiently managed data discovery, lineage, and compliance with all set standards.

Data Loss Prevention: Integrated Data Loss Prevention (DLP) policies ensured that sensitive information was not shared unintentionally outside the organization. This added another layer of protection against potential data breaches.

Sensitivity labels: Using Power BI, we applied sensitivity labels to classify and protect content (reports, datasets, etc.). These labels secure exported data, ensuring its safety even outside the platform, thereby enhancing compliance and data protection.

Feedback loop for continuous improvement: Governance isn't a one-time activity. We instituted regular reviews and feedback mechanisms with stakeholders. This iterative process allowed for the refinement of governance policies, adapting to changing regulatory environments, and business needs.

Incorporating these governance mechanisms, our client was ensured that their vast data assets were managed responsibly, protected effectively, and used ethically. This robust governance framework reinforced our client’s reputation in the fintech industry, laying the groundwork for continued trust and business growth.


Business Outcomes

With our Azure Synapse-based solution, our client’s platform now offers powerful self-service and near real-time analytics, enabling their customers to reach millions of borrowers faster. The platform now has the capacity to scale and support over 1,000 customers. With Azure Synapse, our client can easily integrate machine learning models like fraud detection and recommendation engines without major architecture changes. To accelerate onboarding, we developed an automated deployment framework that onboard new customers in a single click, reducing setup time from days to hours.