Using Microsoft AI and sales data from Dynamics 365, Microsoft Digital built an intelligent sales tool called Daily Recommender to drive richer, more productive conversations between buyers and sellers. Daily Recommender packages buyer insights and recommends next best actions to help Microsoft sales executives prioritize effectively and drive more business. Learn how AI applied to Dynamics data strengthens the buyer-seller relationship and adapts to customer needs by augmenting the seller experience.

Microsoft internal sales executives who manage a large number of accounts operate in a challenging environment. They sell a rich suite of products using an assortment of different sales tools and fragmented data. As a result, they spend too much time gathering and verifying customer information, and too little time helping customers realize how they can achieve their business goals through Microsoft technologies.

Microsoft is hardly alone in this. Distilling compelling insights from disparate, siloed information systems has historically been a complex and time-consuming task for sales executives in all industries. A holistic view of data and insights at the commercial-account level simply hasn’t been available. For sellers, the challenge is that too many tools take too much of their time away from focusing on their customers.

To address these challenges, Microsoft teams within sales, marketing, product groups, and Microsoft Digital partnered to build an AI-enabled sales solution called Daily Recommender. Using Microsoft AI technology, Daily Recommender packages customer insights and recommends next best actions to help Microsoft sales executives prioritize effectively and drive more business.

The evolving role of sales

The modern business-to-business (B2B) buying journey has changed. Buyers no longer depend on sellers to provide basic product information upfront. Instead, customers tend to perform a significant amount of research themselves. According to Forrester research, nearly three quarters of B2B buyers researched at least half of their work purchases online, and 59 percent don’t want to lean on salespeople as the primary source of research.

In most circumstances, customers want to interact with sellers later in the buying journey, when they’re closer to making a decision. Because they’re more informed at this point, they expect a more personalized, productive, and efficient interaction.

To accommodate this shift, the role of the seller must evolve to focus on educating buyers on how they can integrate products into their day-to-day workflows and transform their business. These conversations require deeper technical knowledge from the seller, both of the product suite and of the buyer’s needs. The buyer has done their research, and they expect the seller to have done their research, too.

Historically, that research took the form of preliminary conversations and meetings in which basic customer requirements were established. In this new selling landscape, sellers are expected to be equipped with that information upfront so the initial conversation is a deeper dive into technical requirements, without all the buildup.

The goal for sellers in this new landscape is to remove obstacles and to facilitate a more productive and richer dialogue. To do this, sellers need access to relevant insights before speaking with the customer. They might have downloaded a white paper, attended a conference, submitted a customer-service ticket, purchased a product, or signed up for a newsletter—all actions that can provide insights into the specific problem customers are trying to solve. These details can be difficult to glean in a quick phone call or meeting, and each individual signal can be seen as extraneous and not worth mentioning. By piecing this information together, however, sellers can better understand where their customers are coming from before the initial contact.

The problem is that details such as these aren’t easy to find in the first place, let alone piece together. They’re buried in disparate, often siloed systems. Sifting through these systems and their mountains of data to find relevant insights is a time-consuming task, and one that requires exceptional research skills.

The goal of Daily Recommender is to surface these signals to help sellers respond to and anticipate customer needs while respecting their valuable time. To do that, Daily Recommender augments the selling experience with scientifically generated customer insights and actionable recommendations designed to help the seller identify opportunities for net new business.

Surfacing insights with AI

As illustrated in Figure 1, Daily Recommender processes more than 1,000 data points from more than 15 sources. Analyzing these signals using 20 AI models and a rule-based orchestrator, Daily Recommender provides two types of input to sellers: recommended actions and customer insights.

Recommended actions are actionable and time sensitive. For example, they might alert sellers to create opportunities for net new business (such as new product recommendations) or to potential churn risks.

Customer insights surfaced by Daily Recommender are relevant, self-explanatory pieces of information that, although not directly actionable, provide valuable context and supplementary information to help sellers do their jobs more effectively.

These recommended actions and insights are presented to sellers in Daily Recommender in three components:

• Recommendation. A succinct customer insight or recommended action flagged as high value through the analysis of more than 1,000 signals
• Reasoning and understanding. A real-time explanation of the signals used to generate the recommendation, the timing of the recommendation, and its relevance to the seller’s workflow
• Interpretation and execution. Guidance on how to use the information provided and any supplemental materials that might help the seller take the necessary action

Together, these three components help sellers manage information overload and nurture every phase of the customer relationship with far less effort than before. Sales executives can build these recommendations into their workflows as they see fit. Some sellers choose to act on recommendations immediately, whereas some prefer to “snooze” recommendations until they’ve had adequate time to prepare. Others assign each recommendation to a specific block of time on their calendar, using Daily Recommender to plan out their workweek.

However they choose to utilize Daily Recommender, early results show greater seller impact in three key areas:

• Customer engagement. By reducing time-consuming manual work, Daily Recommender leaves more time for technically deeper, more productive dialogues with their customers.
• Productivity. Less paperwork and less time spent researching and preparing for interactions boosts productivity and makes sellers more efficient.
• Revenue. Daily Recommender users see higher conversion rates and an accelerated sales pipeline.

Ultimately, sellers utilizing Daily Recommender spend less time on high-effort, low-reward interactions, and more time on interactions that deepen the customer relationship and strengthen the sales pipeline. Since implementing Daily Recommender at Microsoft sales centers all over the world, opportunity conversion rates (the ratio of pursuits to sales opportunities) have risen by more than four times, to 30 percent. Past conversion rates hovered between four and six percent.

Expanding seller capabilities with Microsoft technology

These outcomes are accomplished through a comprehensive, holistic overview of customer behavior and intent made possible by Daily Recommender, which has never previously been available to sellers.

Notably, Daily Recommender makes use of both marketing and sales data to paint a complete picture. Because tools used by marketing departments are disconnected from those used by sales teams, the divide between sales and marketing data is a well-known obstacle that both teams struggle to overcome. Both are working with an incomplete understanding of their customers, and there’s little overlap in that understanding between teams. That disconnect causes teams to underperform and can often lead to a jarring customer experience.

Daily Recommender joins marketing and sales, consumption, licensing, and Microsoft Dynamics 365 data into a cohesive whole, often revealing insights that aren’t possible using any of these signals individually.

Recommendations surfaced by Daily Recommender’s machine learning consider four types of inputs: sales data, internal signals, external signals, and relevant business rules:

• Sales data from Dynamics 365, including call frequency and length, lead status, opportunities, win and loss history, and disposition codes.
• Internal signals include Microsoft product usage, licenses owned, consumption of products like Microsoft Azure, Office, and Teams, billed and consumed revenue, workload, and service history. It also includes data from Workplace Analytics and Outlook Graph.
• External signals from Bing Predicts, LinkedIn and other social media, and firmographics account for considerations like company size, location, revenue, industry statistics, PC install base, and industry and SEC codes.
• Business rules gleaned from Dynamics 365, telemetry, and marketing campaigns incorporate product license information, partnerships, and special offer eligibility, for example.

The app is built on the Azure stack and uses the following:

• Azure Data Factory for the data pipeline
• Azure SQL Database and Azure Service Fabric for the web APIs
• Azure Data Lake for the analytics pipeline and data storage
• Microsoft App Insights for telemetry
• Azure Active Directory for authentication

Azure Machine Learning

Each of the inputs provided to Daily Recommender is fed to the sales intelligence AI core housed in Azure. There, the raw data is aggregated and enriched by data connectors, a data wrangling engine, a data conflation engine, and a machine reading engine, as illustrated in Figure 2.

The enriched data is then fed as training data into the machine learning models and into the scoring engine, ranking engine, and recommendations engine. The final output of the scoring, merging, and ranking engines, as well as the output of the machine learning models, is analyzed in Daily Recommender and ultimately presented as a recommendation.

From there, sellers can schedule or snooze a recommendation, mark it as complete, or delete it. Each of these actions is analyzed and fed back to the machine learning models as feedback data or retraining data to fine-tune recommendations further; thus, the application becomes smarter with each use.

Machine learning techniques

Three distinct components contribute equally to the success of Daily Recommender: the underlying AI models, the business context, and precise measurement. Let’s examine each of these more closely:

• AI development. Daily Recommender utilizes a variety of machine learning techniques, including collaborative filtering, random forests, and gradient boosting. Collaborative filtering clusters customers based on similarities such as industry, past purchases, and consumption patterns. This approach recognizes and takes into account the unique needs of each customer while also comparing those unique attributes to the broader dataset. In this setup, a regional hospital that specializes in cancer care will be compared with a hospital of similar size and specialty, not with a larger general hospital. Random forest and gradient boosting, in contrast, look for similarities using a large number of decision trees and produce outputs in the form of product or workload recommendations. The modeling technique that’s ultimately chosen depends on the business context and available datasets. Models are chosen based not solely on outcomes, but on the proper balance of outcomes and business applicability.
• Business AI translation. Equally critical to recommendations is translating the model output to a meaningful business measure. A classification metric, rank, or probability score can be scientifically sound while not providing a lot of value to the seller. To achieve business relevance, two constructs are used. First, all models are trained to a measurable business outcome that is likely to occur in the following six months. The factors that contribute to the business outcome vary. They might include the likelihood of a customer migrating to Azure within the following six months, or the likelihood that a customer activates Microsoft Teams in the same period. Each factor will necessarily be of particular interest to the seller. Next, the model output is translated to a metric (such as conversion rate), measured, and implemented.
• Business measurement. Lastly, an ongoing measurement framework is created. Two dimensions are measured: perceived AI quality and actual AI quality. Perceived AI quality quantifies the value of the recommendations to the seller and the likelihood that sellers are willing to devote time to pursuing the recommendation. Actual AI quality is less abstract, communicating whether or not the desired outcome was reached when the seller did pursue the recommendation. Each dimension yields different insights. Perceived quality might speak to an education or process issue, whereas actual quality tends to reflect issues with the AI itself.

Retraining machine learning models

Daily Recommender monitors sales outcomes over time, using that information to learn and improve the underlying AI models. It uses “supervised learning” to accomplish this, mapping an input to an output based on example input-output pairs to “learn” the resulting function of that pairing. In short, the supervised learning algorithm analyzes training data to produce an inferred function.

In this case, the input is the recommendation surfaced, and the output is the seller’s actions and sales outcomes. Sellers review the recommendations and either accept, reject, or snooze them. Each of these actions has positive and negative connotations that contribute to the supervised learning. For example, deleting a recommendation (indicating that the seller chose not to pursue the recommendation) might counterintuitively have a positive connotation because the product scenario surfaced might have already been discussed with the customer. Similarly, accepting a recommendation (indicating that the seller chose to pursue the recommendation) might have a negative connotation, especially if the recommendation didn’t result in a sale. Supervised learning outputs also take into account the final sales outcome, whether an opportunity was qualified, and whether a sale was ultimately recorded.

Without the translation of business AI and measurement of business outcomes discussed earlier, this learning wouldn’t be possible. Mapping recommendations end-to-end, from recommendation to seller action to sales outcome, creates a standardized supervised learning framework that scales and can be applied to all recommendations. The underlying AI models are continuously measured and retrained based on seller actions and sales outcomes on a monthly basis.

In addition to enabling continuous adjustment and improvements to the AI models, the supervised learning framework provides three key differentiators. First, the framework makes it possible to collect and identify nuances in products and regions. For example, we identified and quantified the impact of “divisional buying decisions” frequently seen in multinational accounts in Asia Pacific. Buying decisions in India, for example, are often driven by decision makers stationed overseas, at corporate headquarters. We also identified an affinity for certain products in certain industries. Both of these insights were subsequently built in to the AI models used via supervised learning.

Supervised learning also gives businesses the flexibility to adjust the balance of quality and volume in their recommendations. Depending on the business context, decision makers might choose to sacrifice quality to increase volume (or vice versa) for a given period. Finally, supervised learning allows for quick reactions to new signals. If a webinar on information security that’s aimed at chief information-security officers has recently gone live on Modern Workplace, and we notice that webinar attendees convert at a higher-than-average rate, the model will “learn” that this signal is predictive and will institutionalize that signal at scale.

Applying the principles of responsible AI

Every project at Microsoft that includes AI adheres to a strict set of guiding principles. The Daily Recommender project is no exception. Here are some of the principles that guided this project:

• Transparency
• Privacy and security
• Reliability and safety
• Accountability
• Fairness
• Inclusiveness

Let’s take a closer look at each of these principles.

Transparency. When a recommendation is presented, contextual information is also provided to the seller explaining the reasoning behind the recommendation, the signals that were used to arrive at the recommendation, and what course of action is being suggested. Rather than being prescriptive, Daily Recommender gives the seller power over the final decision as to whether they pursue the recommendation. This collaborative approach respects seller agency and autonomy.

Privacy and security. Daily Recommender is used at Microsoft sales centers around the world—including New Delhi, Sydney, Beijing, Dallas, Dublin, Fargo, and others—so particularly close attention is paid to local and regional privacy laws such as Europe’s General Data Protection Regulation (GDPR). In general, we apply the strictest data protection regulations to Daily Recommender as a whole. Three dimensions are considered: whether the underlying data within the AI models is compliant with the use case in question, whether the underlying infrastructure is secure, and whether the seller action on the recommendation would also be compliant. Every new feature and release in the application requires a comprehensive privacy and security assessment.

Reliability and safety. Reliability and safety work hand in hand in Daily Recommender. The app provides a reliable and consistent experience for sellers while also taking both seller and customer safety seriously. For instance, if a customer indicates that they prefer not to be contacted via their phone number, that phone number is not displayed in Daily Recommender.

Accountability. In addition to the technical benefits, the business measurement and supervised learning framework provide clear accountability with regard to seller actions and sales outcomes, against which all recommendations are measured. The metrics are integrated into the rhythm of the business, providing clarity on which recommendations are working and where corrective action is needed.

Fairness. Daily Recommender is available to all Microsoft sellers. Given the number of AI models and the diversity of our customer base, we take care to ensure that all AI models meet a certain quality threshold set by the business and the models. The app does not advantage one seller over another. Every seller who leverages the provided recommendations will get the same baseline competitive advantage.

Inclusiveness. The underlying AI models are trained for all accounts, and the recommendations are made available to all sellers. This includes all sales centers, industries, and business sizes.

Built with a modern approach

Historically, a project such as Daily Recommender might have relied on extensive documentation and lists of frequently asked questions to onboard new users as well as possible training sessions. Today, that kind of extensive training is a significant barrier to adoption, particularly in a scaled environment. Instead, we rely on an intuitive, self-explanatory interface that guides users through the experience itself, training them as they use it.

Because of the learn-as-you-go approach, training users on how to use Daily Recommender simply meant encouraging them to use the application. In New Delhi, for instance, sales executives were awarded a certain number of “runs” (a reference to how scores are tracked in the popular sport of cricket) for each task completed in Daily Recommender.

Each week, the five sales executives with the highest number of runs were awarded points that would later be tallied to bestow prizes in a cricket-themed competition. Sales executives became familiar with Daily Recommender in this way. Instead of expecting them to trust Daily Recommender’s recommendations because we asked them to, they began to trust its recommendations because they were using them, and because they worked. The gamification approach drove surges in traffic and established a core mass of early adopters.

The approach for late adopters has been slightly different. Whereas early adopters expected rough edges and the occasional hurdle, late adopters were less forgiving. Driving adoption for this group requires handling objections respectfully, building trust, and then shaping behavior. We listened carefully to questions and concerns raised by late adopters. We worked earnestly to understand their feedback, then identified conflicts, and then addressed them. Frequently, this involved embedding AI-enabled selling into the business and then measuring the impact and sharing success stories.

Best practices

The single most important aspect of the Daily Recommender project—and of any AI project that attempts to improve business outcomes—is marrying the AI to business metrics and understanding the many implications of doing so. Those implications are far reaching and could be related to the sales process, the seller experience, or a business decision.

For example, a rather classic business decision was with regard to the targeting approach for new business and how precise or broad we wanted to be. In the case of Daily Recommender, this means understanding the delicate balance between conversion rates and model outcomes. In some cases, a high conversion rate (the rate at which recommendations ultimately lead to a purchase) is the goal. A high conversion rate might also mean, however, that the sales play supported by the recommendations is very targeted.

Depending on the context, in fact, the conversion rate might be too high, indicating that the potential audience is too narrow. Widening the net would thus bring the conversion rate down, which would affect the accuracy of the AI models being used to generate recommendations. On paper, the lower conversion rate and model accuracy seem to reflect a dip in performance. In reality, however, lower conversion rates with a higher frequency of calls might also translate to more revenue.

Similarly, the experience of sellers who are putting the recommendations surfaced by the AI into practice must be considered and respected. For example, if the target conversion rate is adjusted down too far, that action has a real human toll. To achieve a conversion rate of five percent, sellers would be required to make 100 phone calls to achieve five sales. Sellers would be understandably frustrated with this setup, and the goals of the project would be undermined.

Rather than starting with an AI model, then, each AI model implemented in Daily Recommender began with the sales plays the model was being created to support it. For this reason, we recommend that any similar projects include all stakeholders at the very outset. Data scientists must understand the business context of the project to shape machine learning models accordingly.

Giving equal weight to the business context of a project also means adapting to the speed of business. Business needs don’t conform to the structure of sprint cycles and elaborate planning exercises. They often change from week to week, or even day to day. Planning on a quarterly basis results in a product that is too detached from business goals, so creating AI that supports that speed requires a hyper-agile approach. For Daily Recommender, we adhered to a strict “10/10/10 rule”: 10 hours to define a business problem, 10 days to develop an AI model to address the problem, and 10 weeks to pilot the solution. The agility of the process allowed us to move at the speed of the business, cater quickly to the evolving business priorities, and be an integral part of the business strategy.

Results

To date, sellers have accepted (or acted upon) 60 percent of Daily Recommender’s recommendations, an indication that they are helpful and impactful. Sellers also reported productivity boosts as high as 40 percent.

We also heard from sales executives who lauded the efficiency of Daily Recommender compared to searching through countless Excel files manually for the data they needed to sustain meaningful conversations with customers. One Microsoft sales executive in Sydney, Australia, boasted an increase in pipeline coverage—from 15 to 250 percent—after using Daily Recommender.

What’s next

Daily Recommender has proven remarkably successful at improving seller productivity, helping them reach their sales quotas, and increasing revenue. Daily Recommender processes more than 1,000 data points for more than 32,000 customers across 29 sales plays, which speaks to the ability to deploy more widely in the near future.

More important, Daily Recommender users at each of those sales centers have reported happier customers, thanks to the time savings that can now be spent on serving their customers more proactively and listening more attentively.