illustrated web of productivity icons

Microsoft Productivity Research

Region: Global

2019 Microsoft Productivity Research Collaboration Winners

 

2020 Microsoft Productivity Research Collaboration winners: Mirjam Augstein and Thomas Neumayr

Mirjam Augstein and Thomas Neumayr

University of Applied Sciences Upper Austria

Microsoft lead collaborator: Sean Rintel

Supporting Hybrid Collaboration for the Teams of Tomorrow

Tomorrow’s information workers are increasingly employed in flexible work settings and oftentimes come upon situations where they engage in hybrid meetings and hybrid collaboration. Although such situations, with their dynamic interplay between co-located and remote collaborators, are increasingly supported by software and hardware tools, there are still significant research gaps related to the description and analysis of such settings (which would also allow for more targeted tool support). Thus, the full potential of existing tools such as the Microsoft Surface Hub with its software solutions for co-located (e.g., Shared Whiteboard) or remote (audio and video conferencing) collaboration in the collaborative settings of the future is not yet fully exploited and requires in-depth conceptual as well as technological research. The envisioned research endeavor includes 1) thorough grounding work on a descriptive framework for hybrid collaboration, a small part of which already exists and was published at ACM CSCW (receiving a best paper award) and 2) technical work on a software prototype for the support of hybrid meetings and in-depth (on-the-fly as well as post-hoc) analysis functionalities based on Microsoft hardware and software tools and APIs. To draw conclusions, we will conduct an extensive qualitative user study.

Research engagement results

Workstream 1: Establish models and frameworks that are capable of capturing the distinct nature of hybrid collaboration and meeting activities, building on the grant holders’ prior Domino framework. A systematic literature review was carried out together with colleagues from Aarhus University.

Additionally, territoriality, as an analytic lens often used to inform the design of collaborative systems, was investigated in a hybrid setting. Traditionally, most interactions in co-located settings happen inside one of Personal, Group, or Storage Territories. Data was collected from a range of collaboration groups and several novel aspects of communication channels and access, as well as visibility of certain hybrid territories have been found. These observations led to the definition of a list of 9 design implications.

Outputs:

  • Mirjam Augstein, Thomas Neumayr, Johannes Schönböck, and Carrie Kovacs. 2023. Remote Persons Are Closer Than They Appear: Home, Team and a Lockdown. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Macinery, New York, NY, USA, Article 615, 1–25. (The work reported in this article has mainly been conducted within the scope of the Hybrid Collaboration Spaces (HYCOS) project, funded by the Austrian Science Fund (FWF) [P 34928]. The underlying data was collected during the Supporting Hybrid Collaboration for the Teams of Tomorrow (SHCTT) project, funded through a Microsoft Productivity Research Grant awarded to Mirjam Augstein and Thomas Neumayr in 2019.)
  • Thomas Neumayr, Mirjam Augstein, and Bettina Kubicek. 2022. Territoriality in Hybrid Collaboration. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 332 (November 2022), 37 pages. 
  • Thomas Neumayr, Banu Saatci, Sean Rintel, Clemens Nylandsted Klokmose, and Mirjam Augstein. 2022. Retrieved June 2, 2023, from What was Hybrid? A Systematic Review of Hybrid Collaboration and Meetings Research.

Workstream 2: Partially automate the cumbersome analysis process of collaborative settings (including hybrid ones) and making it accessible to larger audiences. A prototype research pipeline was developed, using AI services to complement and reduce human coding efforts of observational data (e.g., audiovisual recordings and depth information). The resulting ACACIA prototype is capable of recognizing people, objects, and regions of interest, which can also be annotated in an editor. Furthermore, based on the insights gained during the research in the first workstream, to some extent, the closeness of collaboration can be extracted from these data. Future work will focus on further improving ACACIA and bridging the gap to the semi-automated extraction of coupling styles.

Outputs:

The collaboration between MSR and the UAS will continue along the lines of further improving ACACIA and both sides would also like to make use of the opportunity to conduct user studies together in the grant holders’ new 4-year project about Hybrid Collaboration Spaces (Austrian Science Fund P 34928 Standalone Project).

2020 Microsoft Productivity Research Winner: Margaret Beier

Margaret Beier

Rice University

Microsoft lead collaborator: Mary Czerwinski

TeamDNA: Productivity-enhancing Tools for Diverse and Distributed Teams

In this project, we are inspired by the question: what are the fundamental elements that impact team productivity?, for example, what constitutes the “DNA of team productivity?” Due to the necessity and prevalence of teamwork in the workplace, cracking this code and building better teams has the potential to significantly improve workplace productivity and also the teamwork experience. As one would expect, the topic has been studied extensively, especially in Organizational Psychology. However, to date, most prior research has combined human observations with self-reported data, thereby resulting in high-level insights but not deployable systems. Thanks to advances in engineering and Microsoft platforms that enable the real-time tracking of team interactions, we have a unique and unprecedented opportunity to study and improve team processes.

Research engagement results

This project focuses on developing TeamDNA, an automated system that can measure and provide insight into complex and dynamic team processes. Grounded in psychological theory, and enabled by advances in engineering, TeamDNA aims to leverage audio-visual recordings of team meetings to understand key metrics related to individual participation, team dynamics, and the impact of diversity on both individual and team metrics. Results of this work can inform teamwork science and practices for developing more effective teams. The primary two research thrusts are detailed below.

1) Develop novel methods to extract action sequences from interpersonal interactions.

  • Chen, Xu (2021) TeamDNA: Automatic Measures of Effective Teamwork Processes from Unconstrained Team Meeting Recordings. Unpublished dissertation.

2) Establish objective measures that correlate with meaningful psychological indicators of team dynamics, role emergence, performance, and satisfaction within teams.

  • O’Bryan LR, Segarra S, Paoletti J, Zajac, S, Beier M, Sabharwal A, Wettergreen M, Salas E. (Under Review) Modeling Conversational Turn-taking for Insights into Unequal Patterns of Communication Within Multinational Teams.
  • O‘Bryan LR, Oxendahl T, Beier M, Sabharwal A (2021) Communication and the Emergence of Collective Intelligence in Virtual Teams. Remote oral presentation at the Association for Psychological Science Virtual Convention
  • O’Bryan, LR (2020) Approaching studies of collective intelligence in human teams from the perspective of animal swarm intelligence. Remote oral presentation at the ACM Collective Intelligence Conference 2020
  • O’Bryan LR, Beier, M, Salas, E (2020) How approaches to animal swarm intelligence can improve the study of collective intelligence in human teams. Journal of Intelligence, 8(1), 9
  • O’Bryan LR, Cao J, Torres WJ, Sabharwal A, Beier ME (2019) Objective measurement of individual-level communication behaviors underlying interactions within diverse teams. Poster presentation at the 31st Association for Psychological Science Annual Convention. Washington D.C.

For more about this project, contact: Margaret E. Beier

2020 Microsoft Productivity Research Collaboration winner: Jim Hollan

Jim Hollan

University of California – San Diego

Microsoft lead collaborator: Nathalie Riche

A Human-Centered Information Space

For far too long we have conceived of thinking as something that happens exclusively in the head. Thinking happens in the world as well as in the head. Thinking is a distributed, socially-situated activity that exploits the extraordinary facilities of language, media, and embodied interaction with the world. With computers becoming ubiquitous and intertwined with every sphere of life, today we increasingly think with computers. This is accelerated by a radically changing cost structure in which the cost to use a thousand computers for a second or day is not appreciably more than to use one computer for a thousand days or seconds. Yet with all the advances in capacity, speed, and connectivity, using computers too often remains difficult, awkward, and frustrating. Even after six decades of design evolution, there is little of the naturalness, spontaneity, and contextual sensitivity required for convivial interaction with information. We argue that this is a result of a legacy document and application-centered design paradigm that presupposes information is static and disconnected from the context of processes, tasks, and personal histories. We propose a new human-centered view of information: as dynamic entities whose representation and behavior are designed in accordance with the cognitive requirements of human activity.

2020 Microsoft Productivity Research Collaboration winners: Chris North and Doug Bowman

Chris North and Doug Bowman

Virginia Tech

Microsoft lead collaborators: Rich Stoakley, March Rogers

Evaluating Physical and Virtual Large Displays for Windows Productivity Beyond the Desktop

The fundamental space limitations of small display monitors pose significant problems for information workers’ productivity. The increased availability of low-cost, large physical displays and the coming feasibility of virtual displays (viewed through AR and VR headsets) will open fundamentally new user interface opportunities. However, little is known about the value of these modalities for desktop use, the trade-offs between physical and virtual displays, and how to best exploit them for productivity tasks. Our goal is to collect empirical data that will inform the design of future productivity hardware and software, such as Microsoft Windows and Office. These results could help to free users from the confines of current desktop environments and lead to the next major revolution in increased productivity.

Research engagement results

The partnership with Virginia Tech explored how virtual monitors can be used to improve productivity through augmented reality. By looking at range scenarios – from mobile knowledge work to working from home to low vision contexts – the team found ways to expand beyond small screen and monitors that occupy physical space. These videos give an example of the immense flexibility virtual monitors can bring regardless of how much display space is needed:

2020 Microsoft Productivity Research Collaboration winner: Dragomir Radev

Dragomir Radev

Yale University

Microsoft lead collaborator: Ahmed Hassan Awadallah

Improving Employee Productivity Using Text Summarization

Company employees spend a large fraction of their time reading text documents such as company policies, technical manuals, patents, research papers, industry news articles, and email, among others. Reading text takes time that can be used for other work-related activities or for enjoying more leisure time. We are proposing to improve employee productivity, both during onboarding and throughout their entire careers, through automatic text summarization techniques. We will develop a generic, state of the art library, named SummerTime, that will be used on summarization tasks, such as single-document and multi-document summarization, query-based summarization, text simplification, and text re-targeting. The code base will be flexible enough to allow the introduction of new techniques, data sets, and evaluation metrics. We will also implement a number of classic and recent neural algorithms and also improve the state of the art using transfer learning and several novel neural architectures.

Research engagement results

The project with Yale focused on improving productivity via dialogue summarization. Dialogue summarization has become increasingly important since the COVID-19 pandemic given the growth in video conferencing, and the team created and released a benchmark to measure how well current models perform in the dialogue domain.

  • QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (NAACL 2021). Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. Accepted by NAACL 2021. [GitHub, video, poster]

The team also developed state-of-the-art methods for dialog and long-form text summarization. Their results can enable meeting summarization models to help new employees’ onboarding process or new student’s learning process by providing a concise summary of meeting or class interactions.

  • An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next (EMNLP 2021). Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Awadallah, and Dragomir Radev
  • Summ^ N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents (ACL 2022). Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, Rui Zhang
  • DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization (ACL 2022). Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Awadallah, Dragomir Radev

The team also developed a new, wide-coverage summarization library named SummerTime. SummerTime targets non-expert users to expand access for state-of-the-art summarization models to a wider range of people. Users of the library do not need an NLP background and functionality is provided to help identify the best model to use for a particular case, including visualization, automatic model selection, and automatic model assembly.

  • SummerTime: Text Summarization Toolkit for Non-experts (submitted to EMNLP 2021, Demo Track). Ansong Ni, Zhangir Azerbayev, Mutethia Mutuma, Troy Feng, Yusen Zhang, Tao Yu, Ahmed Awadallah, Dragomir Radev [GitHub repository]

For more about this project, contact: Ahmed H. Awadallah

2020 Microsoft Productivity Research Collaboration winner: Akane Sano

Akane Sano

Rice University

Microsoft lead collaborator: Mary Czerwinski

Unobtrusive Personalized Work Engagement Assistant

Work engagement and workload/task management are important aspects of achieving successful and productive everyday information work missions. However, work tasks/schedules and strategies to promote work engagement and well-being could vary from person to person. It is hard to adapt one strategy to all workers. In this proposal, we examine the hypothesis that multi-modal ubiquitous sensors and AI technologies help design a personalized work engagement assistant to suggest personalized productivity management strategies and provide unobtrusive personalized feedback to enhance work engagement and well-being. The aim of the proposed work is to develop and validate an unobtrusive personalized closed-loop system to measure work engagement and workload and provide personalized real-time feedback including work engagement management assistant and subtle sensory feedback based on the user’s physiological and behavioral data. We are focused on the development of unobtrusive and practical technologies and selected the optimal sets of tools and mechanisms based on our team’s interdisciplinary work: ubiquitous and effective sensing and computing, computational imagining, machine learning, organizational psychology, and human-computer interaction.

Research engagement results

The team has developed study protocols to study workers remotely during covid-19 and conducted multiple cohort studies for different types of workers in different locations such as IT workers in the office and at home, hospital shift workers, and caregivers.

The team quantified people’s behavioral and physiological patterns and analyzed the relationships among work patterns, digital device usage, behavioral features such as sleep and mobility patterns, and work productivity, and health and wellbeing outcomes (e.g. mental health, engagement, burnout). Prediction models were developed to infer productivity and health/wellbeing outcomes. The team also implemented and tested the system to run the inference models on the cloud and tested the efficacy of interventions.

Yusuke Nishimura, Tahera Hossain, Akane Sano, Shota Isomura, Yutaka Arakawa, Sozo Inoue, “Toward the Analysis of Office Workers’ Mental Indicators Based on Wearable, Work Activity, and Weather Data“, International Conference on Activity and Behavior Computing (ABC), 2021, [pdf] (Best Paper Award)

2020 Microsoft Productivity Research Collaboration winner: Cyrus Shahabi

Cyrus Shahabi

University of Southern California

Microsoft lead collaborator: John Krumm

Privacy-Preserving Machine Learning Techniques for Improving Individual and Organizational Productivity

Studying patterns of human activity (e.g., moving behaviors, daily routines, organizational workflows) can significantly improve productivity. Neural networks are a powerful tool to capture such patterns, but they need large amounts of individual data (e.g., location data) to train on, which raises significant privacy concerns. This project will design and implement differentially-private techniques to train neural networks. We will focus on skip-grams, which are suitable for sparse data, especially when used in conjunction with negative sampling. We will design algorithms that can build accurate models for human activity patterns, even under strict privacy constraints. We will also study privacy budget allocation strategies across different stages of the model, and we will perform tuning of model hyper-parameters to improve accuracy and performance.

Research engagement results

The collaboration with USC consisted of three different workstreams.

The first effort created:

  • A density-aware technique for publication of OD matrices in a differentially private way,
  • An extension of the OD concept to multiple dimensions that allows one to quantify privately the frequency of certain trajectory segments of interest, and
  • An ML-based technique for accurately answering differentially private range count queries on geospatial data. All techniques were extensively tested using large-scale real datasets and significantly outperformed all existing state-of-the-art approaches.

The second parallel effort identified a new problem of quantifying the intrinsic value of information of trajectories, including a technique for quantifying the intrinsic VOI of trajectories:

The third effort worked on evaluating a methodology for different diseases and transmission models and quantifying the impact of sampling bias. The team considered the spread of SARS and the flu, in addition to work on COVID-19. Results showed the robustness of the method to bias in observed trajectories.

For more about this project, contact: John Krumm