The client
Our client is a Fortune 500 office supply retailer with approximately 1,000 stores across the United States. Store and district managers rely on internal business intelligence reports to monitor sales, performance, and operational metrics.
The ask
The organization wanted to provide managers with an AI chat-based experience that could answer common business questions using data from their internal reports. The goal was to make insights easier to access without requiring users to perform basic calculations manually.
The client initially attempted to build a Microsoft Fabric data agent on their own. However, the agent produced inaccurate and inconsistent answers. The core issue was that the agent lacked clear rules for how to interpret business questions and connect them to the correct metrics in the data model. As a result, the agent frequently created its own DAX calculations instead of using approved metrics, guessed metric names when unsure, and returned unreliable results.
This led to several recurring issues in the agent’s responses:
· Incorrect percentage calculations
· Wrong time filtering, such as double counting date filters
· Inconsistent ranking results
· Discrepancies between goals and actual performance values
· Misinterpretation of business terms and metrics
The objective was to deliver a reliable data agent that could accurately answer common business questions and to implement a repeatable validation process that could be run after each deployment to confirm response accuracy.
Our approach
First, we built a validation framework to systematically test the data agent. The framework accepts a predefined list of business questions along with their expected answers. When the framework runs, it automatically compares the agent’s responses with the expected results. This allows teams to quickly identify errors and confirm accuracy after updates or deployments without manual review.
Next, we refined the instructions given to the data agent. We enforced strict rules that required the agent to use only predefined, approved metrics from the data model. This prevented the agent from creating its own calculations or guessing metric names. We also corrected known issues related to percentage calculations and time-based filtering to ensure results matched existing reports.
To further improve consistency, we introduced a structured data dictionary. This dictionary serves as a single source of truth that clearly defines each metric, its business meaning, and how it should be used. By referencing this centralized dictionary, the agent can reliably map user questions to the correct metrics and business logic.
To speed up testing and improve coverage, we used Microsoft Copilot to generate a wide range of realistic business questions. These questions were then used within the validation framework to test how the agent performed across common manager scenarios.
| Figure 1: Architecture |
The result
The data agent achieved 98 percent accuracy for basic business questions. Store and district managers can now ask questions such as “What were total sales last month?” or “Which product sold the most this week?” and receive fast, reliable, AI-driven answers.
· The agent no longer generates its own DAX calculations
· Responses became consistent and based only on approved metrics
· Hallucinated or fabricated answers were almost entirely eliminated
· Improved accuracy for metrics with similar or closely named definitions
· Stronger alignment between business definitions and reported numbers
· All previously identified functional pain points were resolved
Looking ahead
The organization is exploring opportunities to extend AI chat-based experiences to additional internal workflows, including HR report dashboards. Future enhancements may include support for more advanced and complex business questions.
Ready to lead with AI?
As a Fabric Featured Partner and Frontier AI firm, MAQ Software helps enterprises unlock the full value of Microsoft Fabric. From implementing new data agents to optimizing existing platforms, we support organizations at every stage of their AI and analytics journey.
Contact us at CustomerSuccess@MAQSoftware.com or explore our apps or consulting services on Microsoft Azure Marketplace: