Thursday, November 29, 2018

Case Study: Connecting Databases Across Business Divisions



Key Challenges

   Merge customer identities that are siloed in various databases.
   Create “golden records” that allow division-level databases to identify specific customers.
   Allow administrator access to override data conflicts.

Abundant Data Means Complexity and Redundancy

In ever-changing business environments, systems and databases must evolve to meet increased demands. Business growth inevitably leads to increasing database complexity and often necessitates infrastructure improvements. This new infrastructure requires data to be transitioned and reentered, leaving room for clerical mistakes. As a result, records struggle with inconsistency and redundancy between the various departmental databases. Ultimately, this means that organizations are not able to provide the highest level of service to their customers. Master data management (MDM) is a process that creates a master-level dataset of information, allowing interdivisional communication without the need for costly integration of division-level datasets.

Our client, a large retailer, needed a way to merge customer identities that were siloed within databases in different divisions. The company’s numerous business divisions sold a variety of products. But because they could not track customer identities across business divisions, sales and marketing teams couldn’t coordinate activities. Customers who had just purchased a product still received promotional emails for the very same product. Product sales information was not utilized for future sales and marketing efforts. In some cases, customer records were even duplicated within the same department. The systems hosting the departmental datasets were tailored to the specific needs of each business division, so unifying these systems directly would have been prohibitively expensive.


Our Process

Our client hired us to improve their database accuracy and efficiency. We started by creating reporting copies of each dataset. The next step was to create “golden records”—master customer identities that enabled division-level databases to identify unique customers, despite a multitude of duplicate stored identities.

To create the golden records from the various customer identities siloed within the different divisional databases, we extracted each of the ten divisional databases onto the reporting platform, then cleaned and profiled them. Next, we matched the various records using customized rules, fuzzy lookups, and machine learning procedures. We then deleted duplicate records and standardized and consolidated customer identity values. Machine learning algorithms then identified similar records and linked them, allowing one record to be identified as the master record amongst a group of similar records. We then attributed the master record to the customer identities within the siloed divisional databases. Once mastered, the data was made available for use with the existing systems.

This entire process is now automated and occurs on a recurring basis. Golden records are added and refined to keep pace with changes in the transactional data. While the solution takes advantage of data about customers across various systems to improve understanding of the customer, data security and privacy requirements have been observed. The solution is entirely compliant with the regulations established by the General Data Protection Regulation (GDPR).


Although the process is now automated, we built administrative modules that allow users to address cases where data cannot be properly processed through machine intelligence. Matching rules, thresholds, and other system parameters can also be managed within the administrative module.

Unified Customer Identities Across Divisional Databases

Mastering data from siloed databases allowed our client to clean the raw data, apply business rules to validate the data, and better track customer experiences across business divisions without integrating division-level datasets. The improved data management led to improved customer service. Using our master data management system, marketing and sales can now better address customer needs. Now, customer’s purchases are accurately tracked, and relevant information is sent to them regarding promotions or offers.