Problem

User feedback is needed to assess the system and help it improve over time.

Solution

Implement a user-feedback mechanism that enables the user to provide explicit feedback at each item or instance of system output. The user initiates the feedback interaction.

Leverage user feedback for:

• Evaluating the system and making decisions about deployment.
• Training the system so it learns and improves over time (see Guideline 13, Learn from user behavior, Guideline 14, Update and adapt cautiously, and Guideline 18, Notify users about changes).

Use when

• The system outputs are separate items or instances that can be rated individually.
• The system can leverage explicit user feedback for evaluation and/or training.

How

Decide what type of feedback to collect from the user. Options include:

• Binary ratings (e.g., thumbs up/down; yes/no; like/dislike).
• Ratings on a scale (e.g., 5-star scale, smiley-face scale).
• Thoughts/opinions (e.g., closed-ended short questionnaire, open text-entry field).

Ensure the information collected is both meaningful to the user and can be leveraged for system improvement.

Make the feedback mechanism available for each item or instance of system output.

Consider how to convey to the user that the feedback was received, how it will be used, and whether the user experience changes immediately after giving feedback (see Guideline 16, Convey the consequences of user actions).

Leverage the user feedback data as a metric of system performance (e.g., to evaluate output correctness, utility).

Leverage the user feedback data for system training by working with an AI/ML expert to decide what the model can learn from the feedback and how to integrate this data into the model’s training.

For open-ended feedback that may not be machine-interpretable, put a process in place for interpreting and using the feedback to improve the system.

Consider how to convey to the user that the feedback was received, how it will be used, and whether the user experience changes immediately after giving feedback (see Guideline 16).

User benefits

• Gives the user a voice.
• Supports personalization.
• May improve the experience for all system users.
• Feedback about system performance and user preferences can improve user experience over time (see Guideline 13).

Common pitfalls

• It’s unclear to the user that they can provide feedback to the AI system.
• It’s unclear to the user how to provide feedback to the AI system.
• It’s unclear to the user what the feedback options mean.
• It’s unclear to the user what specific instance or item the feedback is about.
• It’s unclear to the user how their feedback will be used by the AI system.
• In some contexts, such as driving a car, it might be inefficient, dangerous, or disruptive for the user to provide feedback to the AI system.
• User feedback may be subject to reporting bias, where users with moderate views do not provide feedback. Consider mitigating reporting bias by using best practices. If reporting bias is identified in the feedback data, complement it with other data sources for system evaluation and training.

References

• ​​​​​​​​​​​​​​Nan Hu, Paul A. Pavlou, and Jennifer Zhang. 2006. Can online reviews reveal a product’s true quality? empirical findings and analytical modeling of Online word-of-mouth communication. In Proceedings of the 7th ACM conference on Electronic commerce (EC ’06). Association for Computing Machinery, New York, NY, USA, 324–330. DOI:https://doi.org/10.1145/1134707.1134743
• Dellarocas, Chrysanthos, and Charles A. Wood. “The sound of silence in online feedback: Estimating trading risks in the presence of reporting bias.” Management science 54, no. 3 (2008): 460-476.

Notes

• Explicit feedback is information the user intentionally provides for system improvement.
• Implicit feedback is information the system garners from user actions to improve itself.