Evolving Software to be ML-Driven Utilizing Real-World A/B Testing: Experiences, Insights, Challenges
- Paul Luo Li ,
- Xiaoyu Chai ,
- Frederick Campbell ,
- Jilong Liao ,
- Neeraja Abburu ,
- Minsuk Kang ,
- Irina Niculescu ,
- Greg Brake ,
- Siddharth Patil ,
- James Dooley ,
- Brandon Paddock
International Conference on Software Engineering |
Published by ACM/IEEE | Organized by SIGSOFT
ML-driven software is heralded as the next major advancement in software engineering; existing software today can benefit from being evolved to be ML-driven. In this paper, we contribute practical knowledge about evolving software to be ML-driven, utilizing real-world A/B testing. We draw on experiences evolving two software features from the Windows operating system to be ML-driven, with more than ten real- world A/B tests on millions of PCs over more than two years. We discuss practical reasons for using A/B testing to engineer ML-driven software, insights for success, as well as on-going real- world challenges. This knowledge may help practitioners, as well as help direct future research and innovations.