Towards Inclusive Software Engineering Through A/B Testing: A Case-Study at Windows

Engineering software to be inclusive of all those that might/could/should use the software is important. However, today, data used to engineer software can have inherent biases (e.g., gender identity) with inclusiveness concerns. While much attention has been given to this topic in the AI/ML space, in this paper, we examine another data-centric software engineering process, A/B testing, for which we have a dearth of understanding today. Using real-world data from the Windows out of box experience (OOBE) feature, we provide a case-study of how inclusiveness concerns can manifest in A/B testing, practical adjustments to A/B testing towards inclusive software engineering, and insights into ongoing challenges. We discuss implications for research and practice.