Monitor Copilot responses in Dynamics 365 Customer Service 

The Copilot analytics report within Dynamics 365 Customer Service provides a good overview of the usage of Copilot features by agents. You can derive insights into the most frequently leveraged features, impact of Copilot on agent productivity, and agent satisfaction with Copilot responses. While this report provides a good summarization of the impact Copilot has on contact center KPIs, we are seeing interest from customers to drill into individual interaction data, especially around the interactions voted as negative by their agents.

This blog provides an overview of the underlying data that powers the Copilot dashboard, and the additional data points available to you.  We hope that with this context, you can use model customization to modify these reports to better suit your organization’s needs. 

First, make sure your organization is storing agent experience data. You can ask your admin to validate this for you in the Customer Service admin center by making sure they select the Agent experience data checkbox in Copilot help pane. You can continue using Copilot with this setting turned off, but you won’t have historical interaction data to build analytics.

View and analyze agent experience data

When you turn on agent experience data, Dataverse stores the following:

• Transcripts from when agents chat with Copilot
• Agent actions such as copying a summary presented or using a suggested reply from Copilot
• Agent feedback on responses in the form of thumbs up, thumbs down and verbatim feedback

This data is stored in the msdyn_copilotinteraction, msdyn_copilotinteractiondata, msdyn_copilottranscript, and msdyn_copilottranscriptdata tables in Dataverse.

Let’s look at some examples of agent interactions you can view. For instructions on how to download this data, see Download Copilot transcripts | Microsoft Learn.

Interaction 1: Agent asks Copilot a question

The agent asks Copilot “How can I book a trip?” while working on a case. Copilot leverages a knowledge article to generate the following response.

Every interaction an agent has with Copilot is an entry in the msdyn_copilotinteraction and gets a unique interaction ID.

You can use the transcript id stored in msdyn_interactioncontext to get the full transcript of this interaction from the msdyn_copilottranscriptdata entity.

Interaction 2: Agent provides feedback on Copilot’s responses

The agent wasn’t fully satisfied with Copilot’s response and is choosing to rate the response with a thumbs down and provides verbatim feedback.  

Another entry in msdyn_copilotinteraction gets created with the same msdyn_scenariorequestid and msdyn_scenariotype, but with msdyn_interactiontypename set to “ThumbsDown”.

You can use the reference msdyn_interactiondataid to gather the verbatim feedback the agent provided from the msdyn_copilotinteractiondata entity.

Interaction 3: Agent asks Copilot to summarize a case

The agent picks up a case from a colleague and requests Copilot to generate a summary to catch up.  

Another entry in msdyn_copilotinteraction gets created with a new msdyn_scenariorequestid and msdyn_scenariotype set to “Case summary”. 

You can use the reference msdyn_interactiondataid to retrieve the summary that was generated from the msdyn_copilotinteractiondata entity.

Interaction 4: Agent uses Copilot to write an email

Agent decides to leverage Copilot to generate an email with troubleshooting steps using write an email feature. They request an email of type “Resolve the customer’s problem”.  
 
Another entry in msdyn_copilotinteraction gets created with a new msdyn_scenariorequestid and msdyn_scenariotype set to “Write an email”. 

You can use the reference msdyn_interactiondataid to retrieve the email that was generated, along with the knowledge sources used from  msdyn_copilotinteractiondata entity . 

Learn more about monitoring Copilot responses

For more information, read the documentation: Download Copilot transcripts | Microsoft Learn