Data Dictionary for Dynamics 365

Business Scenario

Dynamics 365 is a cloud-based business applications platform. It combines enterprise resource planning (ERP) and customer relationship management (CRM) components with productivity applications and artificial intelligence (AI) features. Out of the box, Dynamics 365 offers diverse tools, but complex business environments often require custom components. These components, called solutions, enable businesses to customize and expand Dynamics 365 features to meet specific business needs.

Each solution is deployed as a single zip file. The zip file contains all the objects (entities, workflows, processes, roles, etc.) present in the environment. The file includes both pre-existing objects and objects that are newly added as part of a current development cycle.

We recently launched a Dynamics 365 implementation for a large software company. The project contains more than 200 entities and several processes and workflows associated with each entity. In Dynamics 365, entities are used to model and manage business data for Customer Engagement apps.

The software company’s Dynamics 365 implementation is still evolving. Sprint-over-sprint we are adding new features and incorporating change requests. These changes include adding new entities or updating or deleting existing entities and their attributes (e.g., field name changes, character length changes, or adding new workflows).

When solutions in large Dynamics 365 projects (such as the project detailed above) are updated, team members must track the changes. Tracking changes allows:

1. The internal team to review and monitor changes before deploying to production.

2. Administrators of downstream systems to anticipate changes that may affect their ETL (Extract-Truncate-Load) jobs.

3. Devs to collaborate on solution updating efforts.

Dynamics 365 does not maintain a subversion of a solution, which would help identify what changed in an updated release. Due to the lack of subversions, it becomes difficult to highlight release over release changes and share them with the larger audience.

We needed to develop a way to keep track of solution changes in a Dynamics 365 environment across multiple releases.

How We Did It: The Technical Implementation

We created a utility called Data Dictionary that accepts two Dynamics 365 solution files as inputs. Data Dictionary compares two versions of a solution (previous and current version). The utility processes the solution files and generates an output in the form of three Excel files. Figure 1 shows Data Dictionary’s comparison flow.

Figure 1: Utility comparison flow diagram

Excel Output Files:

1. Excel file (previous version) - Provides schema of the previously deployed solution.

2. Excel file (current version) - Provides schema of the current solution. This includes the objects that were there in the previous version and those introduced in the current version.

3. Excel file (delta) – This file is similar to the Excel file (current version) but with all the changes highlighted to clearly identify what has been added, modified, or removed from the system in the current version.

Key Highlights:

1. Helps the internal team review (at a glance) what has changed in a solution before deploying the solution to production.

2. Communicates changes between releases to downstream systems. Knowing the changes allows downstream users to update their ETL (Extract-Truncate-Load) jobs, ensuring that they report on the latest data points.

3. Improves dev team collaboration by allowing them to quickly spot differences between two solutions versions.

Business Outcomes:

Data Dictionary saved significant manual effort in identifying and communicating changes in solutions releases. We run Data Dictionary after each major Dynamics 365 deployment and share the output files with key stakeholders.

Engineering Insights: Azure Search-based Web Solution

Business Scenario

A multinational technology corporation offers numerous online learning courses for its premium customers. The customers’ employees rely on the courses to improve their skills, helping them remain competitive in today’s business environment.

The corporation’s previous course management system was cumbersome. The system did not allow employees to quickly search among the thousands of learning courses. Users could only find courses if they knew exactly what they were looking for. Employees needed to be able to browse and filter the catalog and select an offering for their needs. They also expected fast answers. Every millisecond matters!

The corporation needed a scalable solution for future growth.

What customers needed with better search:

1. Better browsing capability – Our client’s employees had difficulty narrowing down courses that fit their expertise level.

2. Enhanced search – Our client’s employees could not search within the course’s content, such as the description or video subtitles.

3. Improved relevance – Our client’s search results did not show the results most relevant to the search term.

4. Rapid free-form text search – Our client needed the ability to support fuzzy search based on terms that have similar construction.

How We Did It: The Technical Implementation

To generate improved search results, we built an entire architecture powered by Azure Search. The corporation’s course details are stored in Azure Cosmos database. Using Change Data Capture, we achieved incremental indexing.

With this architecture, we implemented native search capabilities, faceted filtering, sorting, auto-complete, and pagination in a robust and scalable manner.

By leveraging scoring profiles, we developed a custom solution to improve search ranking and relevance.

Key Highlights:

1. Reduced page load time by 90%.

2. Improved user experience with auto-complete, hit highlighting, sorting, and paging.

3. Created scalable architecture.

4. Incorporated linguistic analysis.

5. Implemented server-side encryption at rest.

Business Outcomes:

With the new Azure Search-powered architecture, courses are no longer undiscoverable. Users are very satisfied with the improved speed and accuracy of results. Users can refine results by technology, video duration, and difficulty level using the rich filtering system. The filtering experience has improved customer confidence, encouraging more users to actively complete the courses--ultimately boosting our clients’ customers’ technical competence.

Unsupervised Machine Learning Model Improves Customer Feedback Analysis

Business Scenario

Our client showcases a variety of technology products in their retail stores to customers. Product demonstrations highlight product capabilities to drive sales. Following product demonstrations, the voice of customer team collects customer feedback. Customer feedback provides a better understanding of how improved product demonstrations improve sales.

To capture customer feedback, the voice of customer team reviewed forms, written comments, reviews, ratings, and surveys using a supervised machine learning model. The team trained the supervised machine learning model using textual data from other teams they worked with. But the supervised machine learning model approach struggled whenever the voice of customer team wanted to add a new dataset. Because the model was based off the provided training materials, the model could not identify material themes that did not originate from the training materials. As a result, each time the voice of customer team worked with a new dataset, the learning model required retraining. To minimize training costs and improve scalability, we needed to create an unsupervised machine learning model.

How We Did It: The Technical Implementation

Figure 1: Unsupervised machine learning model

To create the unsupervised learning model, we researched three topics:

•   Theme identification using supervised models

•   Term frequency-inverse document frequency (TFIDF) methodologies

•   Key phrase extraction

At the beginning of the project, we conducted research to determine how to implement an unsupervised machine learning approach for theme identification. Prior to the completion of our project, no such research had ever been conducted. We started our research with supervised models, even though our solution needed to eliminate manual theme identification. Studying supervised models enabled our team to better understand how human heuristics guided machine learning. To better understand how to track the statistical significance of themes, we studied TFIDF methodologies. By studying supervised model results and TFIDF methodologies, we eventually discovered the key insight we used to structure our unsupervised machine learning model: We could extract key phrases in English using a recurring pattern: (Adj/verb)*(Noun/Proper noun). After discovering the pattern, we worked on several proofs of concept using the pke and rake libraries and Microsoft’s Cognitive Services.

   

The original implementation of the project involved extracting key phrases from customer reviews and feedback comments. First, we divided feedback into different sentences using full stops and spaces. Then, we used several text preprocessing techniques to extract the core message without contractions, punctuation, stop words, conjugations, or spelling mistakes. We then employed text processing techniques to extract the key phrases. Key phrases were identified using nouns and adjectives. (In product feedback, comments typically used the product as the noun and the characteristics of the product as adjectives.) We then used the Word2Vec model to convert the key phrases to vectors. The resulting vectors were then passed through an unsupervised learning engine to cluster the key phrases in order to decipher feedback themes. Each cluster helped identify feedback themes. Finally, we performed a T-test to identify the significant themes. Our approach enabled us to identify newly emerging feedback themes without requiring model retraining.

We fed data to our machine learning model through Azure Databricks. Azure Databricks offered our client the speed and flexibility they were looking for. Azure Databricks allows users to run robust analytics algorithms and drive real-time business insights. Azure Databricks also offers one-click, autoscaling deployment that ensures enterprise users’ scaling and security requirements are suitably met. Azure Databricks also features optimized connectors, which we used to run Microsoft Cognitive Service APIs. These APIs allowed our team to quickly implement entity recognition and key phrase extraction. Because the Azure Databricks solution was managed from a single notebook, our teams could collaborate easily across office locations.

Business Outcomes:

The completed unsupervised machine learning model enabled the voice of customer team to scale the solution to new datasets immediately. Azure Databricks further enhanced scalability and improved deployment speed. With the finished solution, our client can identify themes not contained within training materials and receive significant themes without delay. The completed solution also reduced the manual effort required to identify and map themes.

Infographic: 14 Best Practices for Power BI

Integrated Dynamics 365 Workflow Reduces Manual Effort and Expedites Communication

Business Scenario

Our client displays a variety of technology products to customers in their retail stores. Product demonstrations showcase product capabilities and drive sales. Following product demonstrations, the voice of customer team collects customer feedback. Customer feedback provides insights into how to improve product demonstrations. Improved product demonstrations, in turn, lead to increased sales.

Before we implemented our solution, retail customers provided feedback via phone calls. Our client used a SharePoint system to capture product demonstration feedback. But collecting feedback over the phone and using SharePoint presented a significant challenge. Our client conducted product demonstrations for two different teams without a standardized workflow. The first team, the customer success team, ran product demonstrations to determine how existing customers could use products and offerings. The second team, the enterprise business team, attended product demonstrations to better understand products and capabilities. Both teams needed to coordinate with retail store associates and customers to plan product demonstrations.

To set up product demonstrations, customer success teams and enterprise business teams independently called store associates. No centralized system existed for teams to plan product demonstrations and avoid double bookings. After product demonstrations, customer success team members or enterprise business team members called customers to receive feedback. Because the feedback system was not automated, eliciting customer feedback often required great effort. After receiving feedback, team members manually entered the feedback into SharePoint, which required significant time. Team members could then log in to see feedback for all product demonstrations. Our solution needed to streamline our client’s workflow. A streamlined workflow would deliver two benefits:

•   Reduce manual effort

•   Expedite communication

How We Did It: The Technical Implementation

Recently, Microsoft released a version of Forms Pro allowing Forms to integrate with Dynamics 365 and Microsoft Flow.  We were among the first to implement a solution leveraging integration support between Microsoft Forms Pro, Microsoft Flow, and Dynamics 365. We used the integration support to standardize retail store customer feedback.

We built the pipeline using the Power Platform and Forms Pro and stored customer feedback data in Dynamics 365. We created a Microsoft Flow architecture using the Power Platform. We used PowerApps to capture the details of the planned product demonstrations, generate invitations for the demo events, and capture the feedback in Dynamics 365. Using Microsoft Flow and Forms, we generated the documents to capture the feedback from the demo sessions. Responses from Microsoft Forms route to Dynamics 365, where they are stored under the respective event, customer, or sales leader. We used Microsoft Flow, which is integrated with Azure Active Directory, to create the product demonstration requests.

The completed workflow uses no code. The workflow leverages Microsoft Flow and Forms's service-based back end technology. As a result, the platform is easy to scale and maintain. Workflows are updated by editing or adding templates, avoiding complex code changes. Using the platform, sales leaders create requests for in-store product demonstrations. Store managers then either approve, modify, or reject the request. Forms Pro also enables store managers to share customer feedback. In addition, because customer feedback from Forms Pro is linked to the originating event in Dynamics 365, sales efforts and outcomes are directly attributed to customer feedback.

Business Outcomes:

•   Reduction of manual effort

•   Expedited communication

The integrated solution using Microsoft Forms Pro, Microsoft Flow, and Dynamics 365 improved our client’s workflow at two junctures. First, by changing product demonstration scheduling to an electronic process, we reduced the time needed to schedule product demonstrations. Second, by automating the creation and entry of surveys, we reduced the amount of effort needed to input survey data and improved communication speed. The completed integrated workflow also connected sales outcomes to customer feedback from product demonstrations. 

MAQ Software Releases Award Winning XPRIZE Apps for the Indian Subcontinent in English, Hindi, Urdu, and Bangla

Available in English, Hindi, Urdu, and Bangla, the free Android apps engage children with literacy and numeracy lessons

August 16, 2019, Redmond, Washington: Advancing foundational literacy and numeracy skills, MAQ Software today launched Pehla School, Pehla Class, and Simple Class based on three Global Learning XPRIZE award-winning apps. The highly engaging apps, available for Android tablets and phones, empower children ages 3-10 to improve reading, writing, and math skills.

The apps are available immediately on Google Play:

Pehla School (based on Kitkit School - Global Learning XPRIZE co-winner)

•   Pehla School (bit.ly/pehlaschoolenglish)

•   पहला स्कूल - Pehla School (bit.ly/pehlaschoolhindi)

•   Pehla School - پہلا اسکول (bit.ly/pehlaschoolurdu)

•   পহেলা স্কুল - Pehla School (bit.ly/pehlaschoolbangla)

Pehla Class (based on onecourse - Global Learning XPRIZE co-winner)

•   Pehla Class (bit.ly/pehlaclassenglish)

•   पहला क्लास - Pehla Class (bit.ly/pehlaclasshindi)

Simple Class (based on Chimple - Global Learning XPRIZE finalist)

•   Simple Class (bit.ly/simpleclassenglish)

•   सिम्पल क्लास - Simple Class (bit.ly/chimplehindi)

Our Mission Statement

Over 200 million students in the Indian subcontinent are underserved. While over 95% of children in India are enrolled in schools, many students face significant learning gaps. According to the ASER 2018 report, only half of Class 5 children in India can read at Class 2 level. Only 28% of Class 5 children can do basic Class 2 arithmetic.

With low cost internet access and advances in hardware and software, learning apps are an opportunity to improve learning outcomes worldwide. Google showcased the power of new technology, using advances in Natural Language Processing (NLP) for speech recognition in its free Bolo with Google application. Bolo with Google improves literacy skills in Hindi and English and has been downloaded more than 500,000 times from Google Play.

Based on the open source software developed by the winning teams of $15 million Global Learning XPRIZE, MAQ Software expands free learning app offerings for the Indian subcontinent. MAQ Software optimized the suite of XPRIZE-based learning apps for large scale deployments. Using advances in text to speech implementation in Android phones, MAQ Software will continue to improve the suite of learning apps to utilize new versions of Android-based software and hardware.

By supplementing existing digital tools, school systems and non-profits can improve learning outcomes dramatically through these apps. MAQ Software is providing technical expertise and grants to enable the non-profits to advance their mission.

About the Global Learning XPRIZE

In 2014, the Global Learning XPRIZE launched a $15 million education technology prize. Funded by Elon Musk, the Global Learning XPRIZE challenged participants to develop solutions that enable children to learn reading, writing, and arithmetic within 15 months. Out of the initial pool of nearly 200 applicants, XPRIZE selected five finalists (Kitkit School, onebillion, Chimple, CCI, and RoboTutor) to receive $1 million each.

In field testing, the apps generated remarkable learning outcomes. For example, students who used Kitkit School showed an average reading test improvement of 30–45% and an average math test improvement of 14–37%. Students who did not use the app showed much lower or no change in their test scores.

The Global Learning XPRIZE winning apps help kids: 

Learn on their own: Kids learn reading and math skills through thousands of fun activities.

Personalize instruction: Kids can choose from a variety of reading and math lessons. The apps include hundreds of literacy sessions, math sessions, multi-level books, and videos.

Read without distractions: All the apps work offline without ads, allowing children to focus on the current lesson.

Discover many topics: The apps cover a variety of engaging topics, including animals, science, friendship, and role models.

Showcase creativity: The apps include virtual instruments, a blackboard to practice writing, and drawing and coloring tools to create unique artwork.

About MAQ Software

MAQ Software delivers innovative software solutions to Fortune 500 companies. Using data analytics, cloud technology, and artificial intelligence, we accelerate software initiatives that enable our customers to transform their industries. We help our clients build stronger customer relationships, drive sales, identify new customers, and analyze marketing results from our three state-of-the-art engineering centers located in Washington state and India.

Azure Services Improves Power BI User Experience

Business Scenario

Our client, a large international software company, adopted Power BI across their entire organization. As a result, many internal project managers and sales managers moved their reports to Power BI. The managers’ workspaces utilize hundreds of reports. Because the workspaces contain so many reports, organization is a challenge.

Our client asked us to create a solution for their Power BI reports to solve three challenges:

•   Report organization

•   Filter management

•   PowerPoint creation

The first challenge involved report organization. Because of the way Power BI functions, our client could not divide reports into team subcategories. In addition, team members could not easily organize their own requirements. As a result, team members needed extra effort to locate relevant reports. For our solution to succeed, users needed to find report organization intuitive.

The second challenge involved filter management. Filters applied on one report were not automatically applied to another. Team members could not globally manage filters. As a result, team members who wanted to know about data from France, for example, needed to apply the France filter to each individual report. Only then could team members analyze the data. Before the implementation of our solution, our client used OneNote to document reports for role-specific users. Users then used the documentation to select the correct report and apply the right set of filters. To improve user experience, our solution needed to enable global filter application.

The third challenge revolved around creating PowerPoint decks. Each quarter, business leaders within our client’s organization met to discuss sales and marketing goals. To prepare for the meeting, the business leaders created a PowerPoint using visuals from Power BI reports. In the client’s old process, business leaders needed to:

1. Manually screenshot the report

2. Save the image

3. Move the image to the PowerPoint

4. Format the image correctly

Effort to create PowerPoint slide decks routinely required one month per quarter. The cost of running the VMs each month to generate the slide decks on demand regularly exceeded $10,000. Our solution needed to minimize the VM cost and the time spent creating the PowerPoint for the quarterly meeting. 

To address the three challenges, we implemented a three-part solution:

•   Report collections

•   Global filter panel

•   Export to PowerPoint tool

To improve report organization, we created report collections driven by user roles. Admins can create the collections, and admins can also embed reports on the website. Admins also determine how collections are presented to users. Admins use business capability, focus area, area grouping, role category, and role to determine who sees the Power BI reports. Reports then appear in the appropriate workspace and act as a one-stop shop for the designated users.

To improve filter management, we created a global filter panel. The global filter panel enables central management of all filters across all reports. Individual users can use the global filter panel to customize the filters applied to their reports. As a result, users do not need to reapply filters each time they navigate to a report. Users can also favorite filters.

To reduce the time needed to export Power BI visuals to PowerPoint, we implemented our own Export to PowerPoint tool. Using the tool, users can browse Power BI reports as a gallery of visuals. Once users identify the relevant visuals, they place the visual in an online representation of the PowerPoint slide. Users can choose both out-of-box and custom visuals to include in their slides. When the slides are complete, the tool exports snapshots into a PowerPoint slide deck, complete with custom layout. Our Export to PowerPoint tool reduced the time needed to create the quarterly PowerPoints from one month’s effort to a half hour.

How We Did It: The Technical Implementation

To build the role selection capabilities, we created an Azure web application with embedded Power BI reports. The application connected to an Azure database that housed the report collection and role definitions. The setup allowed admins to add additional reports to the collection through the web interface.

We built the global filter panel using HTML5 controls and integrated the panel as part of the web application. Integrating the panel enabled us to filter selections through the user session and save favorite filter selections. Favorite filters empowered users to customize the default views of the reports to suit individual needs. Global filter selection is passed to the embedded reports, filtering the displayed data within each report.

To generate PowerPoint exports, we used Azure Kubernetes Services to orchestrate two containers. (See Figure 1.) One container loaded the reports in a headless browser (Puppeteer) and captured snapshots of the visuals in the reports. We saved the captured images to Blob storage. We used the other container to generate the PowerPoint file from the selected images and save the file to Blob storage. Once a user successfully downloads the PowerPoint file, the file and images are deleted from Blob storage. 

Figure 1: Export to PowerPoint tool architecture

Business Outcomes:

•   Improved user experience by filtering reports based on role

•   Empowered users to create custom report collections

•   Reduced effort by allowing users to filter reports globally

•   Decreased time required to produce PowerPoint slide decks from days to minutes

•   Lowered costs by implementing a pay-per-use architecture

The completed Azure web application resulted in several benefits for our client. The custom report collections significantly improved user experience. Users no longer had to search to find relevant reports. The global filter also improved user experience. Using the global filter, users could apply the same filters across multiple reports, saving significant amounts of time.

The Export to PowerPoint tool resulted in the most influential benefits for our client. Using the Export to PowerPoint tool, our client reduced the time needed for creating PowerPoints from one month to a few hours. The actual time needed to export the PowerPoint decreased as well, from several hours to five minutes. Most importantly, because the completed Azure web application did not require VMs to constantly run, our client saved over $10,000 per month.