Veterans Health Administration uses mixed reality to provide cutting-edge solutions

Downloads

• Executive summary
• Infographic

Resources

• Medivis - SurgicalAR
• Medivis - AnatomyX
• Verizon 5G
• VHA Innovation Ecosystem
• Blog: Transforming veterans care with HoloLens 2

Products

• HoloLens

The Veterans Health Administration has been a leader in virtual care; last year, 900,000 United States Veterans utilized online healthcare services. As part of the ongoing tradition of enhancing healthcare, VHA has collaborated with Microsoft, Medivis, and Verizon to create the next generation of interactive remote care solutions empowered by advanced technology.

This remarkable partnership has also combined state-of-the-art augmented reality and mixed reality technology with the latest wireless connectivity, cloud computing, and AI to enhance the quality and safety of care for America’s Veterans.

Serving America’s Veterans

VHA is the largest integrated healthcare system in the U.S., with 1,298 healthcare facilities across the country serving over 9 million enrolled Veterans each year. VHA medical centers provide a full range of services including surgery, critical care, mental health, orthopedics, pharmacy, lab, radiology, physical therapy, and extended care.

A new way to provide care

The COVID-19 pandemic accelerated a trend that had been underway for years: treating patients virtually and using digital tools to provide care in new ways.

VHA developed Project Convergence with Microsoft and other partners to use the HoloLens 2 mixed reality headset to change the way Veterans are diagnosed and treated by physicians. The project transforms traditional imaging data, such as MRIs or CTs and converts them into interactive 3D holographic images, opening new possibilities for education, training, and treatment.

A partnership makes it possible

This project required an extensive team of collaborators. “Each partner brought a unique asset to the table,” says Dr. Ryan Vega, Chief Officer for Healthcare Innovation and Learning at VHA. “Medivis can take a medical image and make it appear as if it’s in the real world. Microsoft HoloLens is the hardware that makes it possible. Verizon provided the 5G backbone that allowed moving large amounts of data. And the VHA obviously offers healthcare service to our Veterans.”

Dr. Thomas Osborne is the Director for the VA National Center for Collaborative Healthcare Innovation and is leading the project. He shares that the payoff of the collaboration has been nearly immediate. “Everyone has been absolutely amazed and excited by the technology,” he says. “While using the system, even the most experienced surgeons spontaneously say things such as, ‘Amazing, this is stunning’ and ‘oh now I understand this anatomic relationship.’”

Project Convergence has made important progress in four areas: new educational techniques for medical trainees as well as staff clinicians, high-fidelity interactive 3D presurgical planning, early advancements for improved surgical guidance, and pioneering tools for remote care.

The future of care

Combining HoloLens 2 technology with sophisticated anatomical imaging software from Medivis, AnatomyX is changing the way medical trainees learn about the human body. “The traditional way to begin studying human anatomy is on a flat page of a textbook. However, a two-dimensional image of three-dimensional anatomy is not an intuitive, efficient, or effective way to learn.” says Dr. Osborne. With materials that more closely mirror the actual human form, students can learn the important complexities of anatomy more effectively and therefore have more time to care for patients and concentrate on other medical topics.

Utilizing the augmented and mixed reality HoloLens 2 headsets to train clinicians is just one of the first applications of this partnership’s efforts. Dr. Vega sees multiple ways for another component, SurgicalAR, to help patients now and in the future. “You can imagine residents as well as attending physicians coming together and visualizing the anatomy and talking through a procedure before they're actually in the operating room,” says Dr. Vega. Or a patient can don the HoloLens themselves and take a guided tour of their own body, gaining a better understanding of an upcoming procedure.

Dr. Osborne and his team have already seen the benefits of using SurgicalAR for planning surgeries. For example, when a surgical team was recently preparing for a complex revision of a patient’s knee that had undergone multiple prior treatments, they turned to this new technology. “The anatomy was significantly distorted from all the prior surgeries, and therefore utilizing traditional assessment tools were difficult for everybody,” he explains. Transforming the radiological images into 3D gave the team deeper insight as they worked together to plan for surgery. “There were spontaneous and excited epiphanies—seasoned surgical attendings and their fellows saying, ‘Oh, now I see what's going on! The prior two repairs were done with this approach.’"

In the future, technology like SurgicalAR could even enable remote surgery. “You can really see now how remote surgery can happen,” Dr. Vega says. “An image can be layered over a patient and allow the surgeon to see below the skin’s surface before making an incision,” he says. “A surgeon can be guided by another surgeon halfway around the world, and both see the same images in real-time. I think the benefits of this are immeasurable.”

In addition to education and presurgical planning, Dr. Osborne is excited about how the advanced work of Project Convergence can improve healthcare visits—especially for isolated patients. “One of the great challenges during COVID is the limited ability for physicians and family to be in the same isolation room as a sick loved one. These safety isolation protocols are also important for other conditions as well but can contribute to mental distress. It is exciting that our new and advanced technologies can be leveraged to break down physical barriers by communicating virtually in more profound ways. We can now transport a 3D live hologram of people into another location,” he says. Rather than physically interacting with infectious patients while in bulky gear that hinders personal interaction, physicians and family can utilize volumetric telepresence—what Dr. Osborne calls “holoportation”—to create a more personal experience.

Dr. Osborne sees a future where this technology fosters personal connection not only by overcoming the challenge of physical distance but also by helping physicians focus more on the patient in front of them. “We have multiple impactful projects moving forward in parallel,” he says. One of those is using edge computing and the ability to rapidly process tremendous amounts of data to assist physicians with diagnostic work in real-time. Machine learning and data processing comb through and analyze records far faster than any human, which could lead to a more effective and efficient diagnosis. “And with the right diagnosis,” he says, “you can provide the right treatment, at the right time, so people can have a happier and healthier life. That’s the future of this technology and the types of things that drive us.”