Tuesday, November 20, 2018

Case Study: Boosting Black Friday Sales



Key Challenges

   Create automated emails and a dashboard allowing leadership to analyze real-time sales and promotion data.
   Combine multiple database platforms into a single information portal.
   Implement secure access allowing only authorized personnel to view the sales data.

Black Friday Sales Created Unprecedented Data Volume

Our client is a business strategist in charge of his company’s brick-and-mortar and online stores. His company wanted to coordinate promotional campaigns with near real-time sales performance during the busy Thanksgiving shopping season. We’d worked with our client for many years, and this latest project offered our client and ourselves a real chance to excel. In 2016, we had helped him automate retail reporting, which allowed reports to be delivered every fifteen minutes. Now, our client was tasked with automating the reporting of all sales data—a collection of data that was 100 times larger than the reports we had automated the previous year. With just two months until Black Friday, we hurriedly set to work.

Our Process

From the outset, we knew this project presented unique challenges. Each retail division had a different reporting system. The brick-and-mortar stores relied on custom on-premise databases. But due to the sheer volume of the data we needed to collect, we knew that we could not process and report data at this scale using the on-premise solutions. We would need to use a suite of Azure technologies. These technologies, however, came with their own set of problems. At the time, Azure Analytic Services was not yet out of preview. Many of the features we needed lacked documentation. With these unknowns in mind, we conducted a series of proof of concepts over a period of three weeks. We then vetted our assumptions, communicated the results with our client, and determined our final architecture design. At the time, no one had hosted a project of this scale within Azure services before. We were working in uncharted territory, but our client had complete confidence that we would deliver.

We began implementation immediately after conducting our proof of concepts. We initially had some challenges getting data at the rate we needed. We had access to real-time sales reports from retailers and online stores but did not have access to these reports from two specialized business divisions. “A lot of my time was spent reaching out to folks who managed that data,” our project lead observed. “Once their teams saw the initial version of the build, they were willing to give us access to the transactional data because they understood how successful they could be with our implementation.” Once again, our Agile methodology proved crucial throughout this entire process. Because we submitted deliverables immediately upon completion and finished the implementation ahead of schedule, we were able to win the trust of both specialized business divisions in the span of a few short weeks, despite having never worked with them before.


We finished development and basic testing two weeks before Thanksgiving. We continued running end-to-end tests improving our implementation until the Wednesday before the holiday. Our teams were absolutely committed to the successful closure of the project: “The weekend before Thanksgiving, the entire team and I worked very hard. I don’t know how much sleep I got those five days, but I can tell you this: I was awake most nights,” our project lead said with a laugh. That week, our teams busily worked, simulating the data reporting load of the Thanksgiving shopping season. Finally, it was time to put our implementation to the test.

The first three days of the holiday weekend passed without a hitch. But on the fourth day, a minor setback arose. Because we were pulling the data in near real time, we were creating a lot of files. We were creating roughly fifty files every thirty seconds, and the data reporting job started running slower and slower. Initially, this was not a problem, but when the job started taking over twenty-five minutes, our engineering team stepped in. If the data reporting job took over thirty minutes, it would eat into the timeframe allotted for the next data reporting cycle. This would have then disrupted all the reports.


Our team first implemented a quick fix that compiled the files as they came in. This fixed the setback for a day. By the next day, the team reached a permanent solution; files were defragged and only then compiled.

Improved Sales Reporting Without a Hitch

Our work resulted in a tremendous triumph for our client. Despite the technical challenges we faced during the Thanksgiving festivities, there was no impact on end users. Each of the 125 email reports was sent to promotional staff on time. Our client made five promotional decisions based on the reporting that we did, and each decision resulted in improved revenue results. Our reports were widely used; the dashboard that collated the sales reporting data ranked 100 out of 75,000 dashboards used by the organization.