Watch For

Project Lookout was developed as an app to monitor live video streams on behalf of a person and notify them when specified events occur. Such an idea can be very powerful using artificial intelligence with many different applications. The five team members who came together during Microsoft’s 2017 Hackathon were all from Microsoft Research, and two of them had been analyzing traffic camera footage using AI as part of their day job. The team decided to apply the same idea to a consumer scenario for their hackathon project, to analyze live streaming footage and alert the viewer if something interesting happened.

Leaders around the company saw the potential of the project and they were so impressed that the team was awarded the 2017 Hackathon grand prize for Lookout.

Next steps for the team was preparing themselves and Lookout to receive their prize – a meeting with Microsoft’s CEO, Satya Nadella. Garage team members worked intensively with Lookout to help them create their value proposition and pitch, as well as to plan their first experiment. The project also got a new name – Watch For.

Watch For caught the attention of Chad Gibson, general manager of Mixer. A live-streaming platform with a gaming focus, Mixer found themselves with a much larger volume of streams than anticipated. As Gibson notes in a Fast Company story, “And so we were trying to find, ‘How can we provide new, unique ways of allowing players of PlayerUnknown’s Battlegrounds or Fortnite to be discovered?’”

Mixer saw the potential of the Watch For project to help solve this problem using artificial intelligence and computer vision.

Once Mixer reached out to the team, Lenin Sivalingam, Matthai Philipose and Peter Bodik from the original project began iterating and incubating on possible ways Watch For’s AI analyses could be implemented to help Mixer surface the most interesting parts of their gaming communities’ livestreams. A few months later, the scrappy team had a prototype ready for Mixer to implement.

Watch For supplied backend algorithms and trained machine learning models on PUBG game elements on screen, “watching for” things like text on screen, icons, player stats and score. Watch For’s large-scale video analysis of every stream coming into Mixer became the technology behind Mixer’s successful HypeZone.

HypeZone channels show the last – and most exciting – rounds of livestreaming battle-royale type games like Fortnite where players compete to become the last one standing. With the addition of HypeZone, Mixer saw a significant increase in viewers, and more activity around streamers who wanted to be featured on HypeZone by streaming more often and upping their game. Over time the Watch For team and their technology has evolved to understand more varied content.

Soon after Watch For’s partnership with Mixer took flight, a team from Bing also reached out to see how Watch For can be leveraged in search. Bing saw this as an opportunity to add more esports coverage and to surface robust esports search results. Professional gaming has become a growth industry, with higher stakes, brighter fame for players, and an exploding number of fans, but the data that powers statistics and analytics is still highly fragmented. This fragmentation makes it difficult for fans and players to find the information they are interested in and also makes it difficult for events, teams, and websites to create engaging experiences for their users.

The Bing esports team believed that, by leveraging Watch For, they could bring together a unique and comprehensive pool of data that spans most popular esports at both the professional and amateur levels. Their first experiment was to use Watch For to provide richer search for streamers. The existing search function on popular streaming sites such as Mixer and Twitch only allow searching by game and by number of viewers. This causes popular streamers to become even more popular while new and upcoming streamers struggle to show up in search results at all. By using Watch For, Bing can now show results not just by popularity, but by other factors like number of players left in game and elimination count. Watch For allows Bing users to discover new streamers based on their talent as well as their popularity.

Watch For is helping Bing esports accomplish three major objectives: Democratizing the streaming ecosystem by allowing streamers to be discovered for more qualities than viewership alone, bringing together all the best data regardless of what level of player you are – from fan viewers to casual gamers to pro – and, making Bing the best place for esports experiences.

“Watch For is a key differentiating technology.” Donald Brinkman, senior program manager on Bing esports. “It opens up entirely new approaches for how we can present esports content to our users in a way that’s really personalized to exactly what they’re looking for.”