Microsoft Research 2019 reflection—a year of progress on technology’s toughest challenges

Research is about achieving long-term goals, often through incremental progress. As the year comes to an end, it’s a good time to step back and reflect on the work that researchers at Microsoft and their collaborators have done to advance the state of the art in computing, particularly by increasing the capabilities and reach of AI and delivering technology experiences that are more inclusive, secure, and accessible. This covers only a sliver of all the amazing work Microsoft Research has accomplished this year, and we encourage you to discover more of the hundreds of projects undertaken in 2019 by exploring our blog (opens in new tab) further.

Improving the reach and accessibility of AI and machine learning

Machine learning has made a tremendous impact on people’s everyday lives, especially in the latter half of this decade, while also raising significant policy and societal issues for research to address. This year, Microsoft researchers and their collaborators worked to improve the capabilities of machine learning systems and also explored new models that can take the discipline further. They used unique approaches that can make these systems more accessible and inclusive.

In deep learning, Jianfeng Gao (opens in new tab)’s team released MT-DNN, (opens in new tab)a model for learning universal language embeddings that combines the strengths of multi-task learning and the language model pre-training of BERT, helping systems quickly develop the semantic understanding necessary for natural language processing. And Xu Tan (opens in new tab) and his collaborators at Microsoft Research Asia (opens in new tab) developed MASS (opens in new tab), a pre-training method that outperforms existing models at sequence-to-sequence language generation.

In the coming years, breakthroughs in machine learning will emerge by exploring new models beyond the current foundation of using Markov decision processes, particularly as reinforcement learning—a data-hungry approach generally suited to simulation scenarios—becomes more applicable to real-world scenarios. In this podcast (opens in new tab), John Langford (opens in new tab)and Rafah Hosn (opens in new tab) discuss these new directions in reinforcement learning and their applications to everyday computing, including the real-world RL now deployed in Personalizer, (opens in new tab)an Azure Cognitive Service. Langford and Alekh Agarwal (opens in new tab) also hosted a webinar (opens in new tab) on the foundations of real-world reinforcement learning.

Many machine learning applications benefit from training with very large datasets; however, many potential uses simply do not have enough data for typical approaches to be effective. Enter machine teaching, where domain experts can build bespoke AI models with little data—and no machine learning expertise. In this podcast (opens in new tab), Riham Mansour (opens in new tab) discusses (among other things) LUIS, one of the first Microsoft products to deploy machine teaching concepts in real-world scenarios.

Another project aimed at further democratizing AI is the Decentralized & Collaborative AI on Blockchain framework (opens in new tab) with Justin Harris, (opens in new tab)which enables users to train and maintain models and datasets on the Ethereum network. At NeurIPS 2019, Debadeepta Dey (opens in new tab) and collaborators presented Project Petridish (opens in new tab), an efficient forward neural architecture search algorithm that helps identify suitable neural architectures for a given machine learning task. And Adith Swaminathan (opens in new tab) and Emre Kiciman (opens in new tab)’s February blog post (opens in new tab) explores researchers’ work to improve causal inference modeling, which helps AI better understand “what if” scenarios in a wide variety of contexts.

Enabling responsible, inclusive, human-centered innovation

2019 kicked off with the inaugural ACM FAT* Conference in Atlanta, which focused on fairness, accountability, and transparency in socio-technical systems. Microsoft Research presented four papers (opens in new tab) at the conference. They covered gender bias in occupation classification, the role of data-driven decision making in reinforcing or amplifying injustices, strategic manipulation of algorithmic decision systems, and the fair allocation of items in scenarios without money, respectively. This work came from the FATE research group (opens in new tab) at Microsoft, which studies the complex social implications of AI, machine learning, data science, large-scale experimentation, and automation.

At May’s CHI Conference on Human Factors in Computing Systems, Saleema Amershi (opens in new tab) and her collaborators presented a set of guidelines for human-AI interaction design (opens in new tab) that brings together more than 20 years of research, recommendations, and best practices around effective interaction with AI-infused systems. Bringing this work together will help designers manage user expectations, moderate the level of autonomy, resolve ambiguity, and provide users with awareness of how systems learn from their behavior.

To ensure that machine learning systems effectively do the jobs we deploy them to do, we must develop a deeper understanding of how they succeed and fail. This paper, (opens in new tab)from researchers at Microsoft Ram Shankar Siva Kumar (opens in new tab)and Jeffrey Snover (opens in new tab)and their collaborators at Harvard, articulates the various ways machine learning systems can fail—either through intentional adversarial attacks or unintentional failures in which the output is formally correct, but unwanted.

Helping to train autonomous systems that can be trusted in real-world applications, the open-source simulator AirSim (opens in new tab) provides realistic and detailed testing environments. This year, it played host to the NeurIPS competition Game of Drones (opens in new tab). In the drone race challenge, participants competed against a Microsoft Research opponent on the same track, working with a level of strategy and maneuvering not generally offered by contests of its kind. Microsoft researchers and collaborators who organized the competition plan to keep it open and add new racing environments. Visit the GitHub repository (opens in new tab) for more information.

In January, Jenn Wortman Vaughan (opens in new tab) and Hanna Wallach (opens in new tab) hosted a webinar on Fairness in Machine Learning (opens in new tab), demonstrating how to make detecting and mitigating biases a first-order priority in the development and deployment of machine learning systems.

Creating human-computer interaction that works for all

This year, Microsoft researchers continued their work to make computing more natural, comfortable, and accessible for everyone. At the ACM CHI Conference on Human Factors in Computing Systems, researchers presented a number of papers and demos (opens in new tab)exploring how to support accessibility for users with cognitive or sensory disabilities. These include studies on whether web browsers’ “reading mode” is truly helpful for people with dyslexia and tools to help VR be more accessible for people with low vision (including tunnel vision, brightness sensitivity, and low visual acuity).

Also presented at CHI was Microsoft Soundscape, (opens in new tab) a project that uses 3D audio cues to enhance situational awareness and assist with navigation. (You can try the app yourself here (opens in new tab).) In this op-ed (opens in new tab) in the Toronto Sun, Microsoft researcher Bill Buxton elaborates on the importance of work like this, noting that 1 billion people worldwide have some form of disability, making it imperative that we create technologies that support personal autonomy.

Speaking of sound, Nikunj Raghuvanshi’s (opens in new tab) podcast (opens in new tab) explores the physics of audio and discusses Project Triton (opens in new tab), an acoustic system that models how sound waves behave so that the audio in 3D game environments can be as rich and immersive as the graphics. Project Triton (opens in new tab) is available (opens in new tab) for any game via the Unity and Unreal game engine plugins, as part of Project Acoustics (opens in new tab), powered by Azure.

At the ACM Symposium on User Interface Software and Technology, Microsoft researchers presented a number of projects (opens in new tab) that make virtual environments more realistic, tactile, and navigable. Dreamwalker (opens in new tab) is a VR project that can augment a real-world walking experience with virtual reality—a virtual environment that detects the user’s surroundings in real time and generates a virtual world that accounts for their path and any obstacles—so that you can walk to work in Seattle, but through a virtual Manhattan. Mise-Unseen (opens in new tab) is a project that uses gaze detection to modify or replace elements of a virtual world while the user’s attention is directed elsewhere. And CapstanCrunch  (opens in new tab)is a VR controller that leverages centuries-old technology, once used to control ropes on sailing ships, to provide effective and inexpensive haptic feedback.

Meanwhile in the physical world, Microsoft researchers partnered in May with students at the Brooklyn Public Library’s Fashion Academy to weave technology into their designs (opens in new tab) using Project Alava (opens in new tab), which aims to develop microcontroller-based systems that are simple to build and code for people with a limited computer science background. At their end-of-program fashion show, students showed garments that incorporated LEDs, motion sensors, and motors. You can read about other areas where Microsoft researchers are working at the intersection of art and science here (opens in new tab), including Ada (opens in new tab), a first-of-its-kind architectural pavilion that incorporates AI, on display at Microsoft Research Redmond. (opens in new tab)

Breakthroughs in security, storage, systems, and applications

2019 saw continued progress in the development and adoption of homomorphic encryption, which enables computation on encrypted data, helping to preserve privacy. Microsoft SEAL has become one of the world’s most popular homomorphic encryption libraries, with broad adoption in both academia and industry. In February, Microsoft took the next step in democratizing homomorphic encryption by releasing SEAL for .NET (opens in new tab). (The Microsoft SEAL library is open source and available (opens in new tab)on GitHub.)

In August 2019, Microsoft researchers joined their industry and academic peers for the Homomorphic Encryption Standards Meeting. The group will reconvene at Microsoft Research in Redmond for next year’s meeting this February. Take our webinar (opens in new tab) to learn more about homomorphic encryption, and listen to this October podcast (opens in new tab) with Craig Costello (opens in new tab) for an overview of the year’s developments in cryptography generally, including efforts to prepare for a post-quantum future.

In April, Project Everest (opens in new tab) took another step forward in its work to build a verified, secure HTTPS ecosystem with the release of EverCrypt (opens in new tab), the first fully verified cryptographic provider to meet the security needs of the TLS protocol. Project Everest (opens in new tab) is a collaboration between Microsoft, Inria, and Carnegie Mellon University.

By 2023, it’s expected that more than 100 zettabytes of data will be stored in the cloud. To meet that need, Project Silica (opens in new tab) is developing the first-ever storage technology designed from the media up for use in the cloud. This year, the team collaborated with Warner Bros. on a proof of concept, storing the 1978 film Superman on a nearly indestructible piece of glass roughly the size of a drink coaster. This work is part of the Optics for the Cloud Research Alliance, which you can learn more about here (opens in new tab)or on the Microsoft Research Podcast (opens in new tab). Meanwhile, researchers at Microsoft and the University of Washington achieved a “Hello, World!” moment in April for a different way to meet our growing storage needs: They demonstrated the first fully automated system to store and retrieve data in manufactured DNA (opens in new tab). (For more on the intersection between computing and biology, listen to this podcast (opens in new tab) featuring Andrew Phillips (opens in new tab), who leads the Biological Computation Group (opens in new tab) at Microsoft Research Cambridge. (opens in new tab))

Cambridge researchers Andy Gordon (opens in new tab) and Simon Peyton Jones (opens in new tab) demonstrated the practical impact of fundamental research by exploring how ideas from programming language research could improve one of the world’s most common business applications: the spreadsheet. In this January blog post, (opens in new tab) they detail how their collaboration with the Microsoft Excel team led to product improvements such as cells that can contain first-class records linked to external data sources and formulas that can compute array values that “spill” into adjacent cells.

At the ACM International Conference on Web Search and Data Mining, Microsoft researchers presented new work in extreme classification, a research area that promises to dramatically improve the speed and quality of algorithms that can answer multiple-choice questions involving uncertainty, where there could be multiple correct answers. Among other things, this work can lead to more relevant recommendations and search results. In this blog post (opens in new tab) from February, Manik Varma (opens in new tab) of Microsoft Research India (opens in new tab) provides a deep dive into extreme classification.

Thanks to gains in computer vision, particularly object detection and classification, video analysis has become far more accurate; however, fast and affordable real-time video analysis is lagging. In December, Microsoft researchers Ganesh Ananthanarayanan (opens in new tab) and Yuanchao Shu (opens in new tab) hosted a webinar (opens in new tab) on Project Rocket (opens in new tab), an extensible software stack that leverages the edge and cloud to meet the needs of video analytic applications.

In April, the Microsoft Research Podcast turned its attention to databases—particularly the need for imperative programming that allows for good software engineering practices like modularity, readability, and reusability. In this episode (opens in new tab), Karthik Ramachandra (opens in new tab) discusses Froid (opens in new tab), an extensible and language-agnostic framework for imperative functions in databases, which is available as “Scalar UDF Inlining” in Microsoft SQL Server 2019 (opens in new tab).

Open-source tools and data for the research community

Throughout the year, researchers from Microsoft made a number of projects open source for the benefit of the academic community, including the following:

• • SandDance (opens in new tab), a data visualization tool included in Azure Data Studio, Visual Studio Code, and Power BI
  • TensorWatch (opens in new tab), an AI debugging and visualization tool
  • PhoneticMatching (opens in new tab), a component of Maluuba’s natural language understanding platform
  • SpaceFusion (opens in new tab), a learning paradigm that brings together a palette of different deep learning models for conversational AI
  • Icecaps (opens in new tab), a toolkit for conversation modeling
  • Icebreaker (opens in new tab), a deep generative model that minimizes the amount and cost of data required to train a machine learning model

Building on last year’s announcement of Microsoft Research Open Data (opens in new tab)—an Azure-based repository for sharing datasets—the company developed a set of data use agreements, released them on GitHub, and adopted them for a number of public datasets (opens in new tab). This work aims to make research data more readily available in the cloud and to encourage the reproducibility of research.

Supporting and honoring the research community

This year, Microsoft Research introduced the Ada Lovelace Fellowship to support diverse talent from underrepresented groups pursuing doctorates in computing-related fields. You can read about the fellows and their research here (opens in new tab). Ten doctoral students were also awarded two-year fellowships (opens in new tab) as part of the PhD Fellowship program, supporting research in photonics, systems and networking, and AI. Additionally, Microsoft Research awarded Microsoft Research Faculty Fellowships to five early-career faculty members pursuing high-impact breakthrough research. You can read about their work here (opens in new tab).

A number of researchers at Microsoft received awards and honors throughout 2019—check out the full list of recipients here (opens in new tab).

Finally, we are saying goodbye to Harry Shum (opens in new tab), who is leaving the company in February after 23 years, and hello to Microsoft CTO and EVP Kevin Scott (opens in new tab), who has assumed Shum’s responsibilities as head of the Microsoft Artificial Intelligence and Research Group. Listen to Scott on the Microsoft Research Podcast here (opens in new tab).

We hope you had a good year, and we look forward to a 2020 full of collaboration and exciting breakthroughs. Happy holidays.

To stay up to date on all things research at Microsoft, follow our blog (opens in new tab) and subscribe to our newsletter (opens in new tab) and the Microsoft Research Podcast (opens in new tab). You can also follow us on Facebook (opens in new tab), Twitter (opens in new tab), YouTube (opens in new tab), and Instagram (opens in new tab).

Opens in a new tab