Microsoft Envision

This article is part of Microsoft Envision, a digital series exploring critical business and technology trends. 

The new (mixed) reality for healthcare

Lawrence Crumpton, Mixed Reality Solutions & Partners Lead ANZ, India, Korea & South East Asia and Simon Kos, Health Industry Advisor 

Ageing populations, rising chronic illness, nursing shortages – healthcare systems around the world are straining under the weight of these complex problems. But technology is revolutionising clinical delivery and training by extending connectivity and AI into the hands, and eyes, of medical professionals. 

What makes mixed reality a game changer? 

Mixed reality is often thrown into conversation alongside virtual reality and augmented reality, but they aren’t synonymous. VR is about being isolated and immersed in an experience with no real-world input. AR overlays information over the real world, but it’s offered up without interpretation or translation – like a doctor looking at an x-ray.  

Mixed reality, or MR, takes it up a notch. This is a ‘heads up, hands free’ experience utilising a device, like our HoloLens 2, that lets you use voice, touch, gesture, sight and hearing to make real-time use of things like AI, holograms, soundscapes, collaboration, natural language processing and translation, context-driven insights and a whole range of interactive tools.  

While these innovations aren’t limited to healthcare, there are good reasons why this technology is profoundly applicable to this field. Firstly, quality of care is one of the biggest reputational risks to health organisations. MR can help improve outcomes – that is, de-risking delivery – by providing oversight and clinical decision support. Countless healthcare organisations have highlighted to us that they are ‘drowning in data.’ On top of this, medical knowledge doubles every few months, making it extraordinarily difficult for practitioners to stay at the forefront of their field. So being able to turn data into knowledge and then feed it to people when they’re with patients is the next frontier, and this is the technology to do it. 

Is MR really transforming healthcare? 

Around 2015, MR in healthcare exploded. From patient consults to training, collaboration and assisted surgery, MR has found a home in the healthcare ecosystem. And as the tech improves, the opportunities expand. 

MR has been transforming training for quite some time, with its ability to provide holographic step-by-step guidance without subject matter experts needing to be physically present. For example, it has improved comprehension of anatomy so dramatically at Case Western Reserve University that the school is now using it as the primary anatomy education tool for undergraduates – instead of cadavers. They found that the first cohort of students who trained using MR had up to 80% better information retention. 

MR also enables collaboration in an unobtrusive way. Medical staff can connect to experts remotely and pull them into an interactive collaboration. And a HoloLens device handed to a patient allows them to explore holographic 3D visualisations of their internal systems and proposed treatment, making procedures easier for clinicians to explain. 

Surgical teams in operating rooms can use Dynamics Remote Assist to connect themselves instantly to specialists and colleagues in real time, whether in a lab on the other side of the hospital or in a specialist centre across the world. Their colleagues act as a second set of eyes, provide expert analysis and even share patient data or images for the surgeon to be able to access in an unobtrusive way. 

An exciting emerging research field is around intra-operative surgical guidance. In accordance with local regulations, many health systems are conducting clinical research in the use of MR as a reliable method of intra-operative guidance. For example, Dr Gao Yujia and his research team at National University Health System in Singapore are using MR during brain surgery to project guidance right onto the patient’s head, showing the surgeon what’s happening underneath the skull in real time. 

NUHS is exploring how digitally assisted procedures could one day give neurosurgeons an enhanced and augmented 3D view of the patient’s brain by using using the MR capabilities of Microsoft HoloLens 2. Currently, doctors look up at a screen then down at the patient – they are in effect mentally reconstructing a 2D image (i.e., a scan) onto a 3D object (the patient). This constant mental back and forth is not easy in a high-stress environment. The aim is for HoloLens 2 to replace this with a 3D image of the brain displayed on the headset visor, superimposed over the patient’s real-life form. The image is locked in position, so even if the surgeon walks around the operating room they see the image accurately. In fact, multiple surgeons wearing headsets in the same room can share the same experience. 

MR can build resiliency 

The past few decades we’ve all put a lot of effort into digitising records. While electronic health records (EHRs) bring notable benefits, there’s an elegance and simplicity to pen and paper that’s been lost to the point where the clerical burden introduced by EHRs has become a leading cause of physician burnout.¹  

Mixed reality is a huge opportunity to move from what is essentially a mechanical input to a natural user interface that brings back that elegance and simplicity. This helps healthcare organisations build resilience by freeing up workers to focus on clinical care.  

Health records are just the tip of the iceberg for MR. Technologies like natural language processing with AI could help doctors automatically create medical records and referrals while they talk with a patient, saving time and reducing admin.  

On the front line of healthcare, MR helped a UK hospital reduce risk and keep more healthcare workers on the job during the pandemic. Imperial College London used HoloLens headsets to do virtual ward rounds, which meant only one member of staff was potentially exposed to Covid-19 while the rest of the team could still participate virtually. 

In Taiwan, they’ve extended the practice to community providers who are using HoloLens to bring virtually any specialist with them into people’s homes. 

Training our next-gen professionals 

Nursing shortages are set to worsen if solutions are not found. We’re excited about the possibilities MR offers to help train new nurses. Crucially, MR allows you to expose students to the ‘hard stuff’ – letting them learn from making a critical care mistake in a virtual training environment.  

We created interactive holographic patient simulations, so as students make care decisions, these ‘patients’ will deteriorate or get better right in front of their eyes. It’s how it works in the real world, but here they get to learn those patient management skills safely. And with that confidence and experience, they can hit the ground running.  

By making better use of AI through the conduit of MR, we can ensure that critical information is provided to the right people at the right moment. In short? We now have the power to reduce cognitive load so we can increase the focus on what really matters – which, in healthcare, makes a life-or-death difference. 

For more on critical technology trends, visit envision.microsoft.com