Skip to main content
Applied Sciences

Project Helipad: a holographic interface

A user interface in the space around you

The illustration shows a laptop with a hologram applied to the keyboard and a virtual interaction plane an inch or two above this, a 3D sensor positioned between the display and the keyboard to monitor the user's eye location, and a light source above the display shining on the hologram.
Schematic of the Project Helipad demonstrators discussed in this article

Typical user interfaces in use today, especially touch interfaces, exist on two-dimensional displays or physical surfaces such as a computer monitor or a keyboard.  Project Helipad explores extending these interfaces to the full three-dimensional volume accessible to the user, which provides more space for UI elements.

The ability to make use of vertical space would be particularly helpful on devices with limited surfaces, such as a tablet with a small keyboard attached.  On a standard keyboard, keys are of a standard size and spacing that users are already accustomed to.  Attempting to maintain those dimensions with the typical set of keys on a comnputer keyboard quickly fills up a small mobile keyboard, leaving little room for other UI elements such as a trackpad.

What if we could “place” a trackpad (or other UI element) in the 3D volume above a keyboard?  To do this, we need to solve a couple of problems.  First, somehow we must give the user a visual indication of where in that 3D space the virtual trackpad is positioned.  Secondly, we must measure when and how the user has interacted with the virtual trackpad.  In this project, we explored using holograms as a floating visual indicator, which we integrated with a touch-sensitive display and with a force- sensitive keyboard, and we used depth-sensing systems, such as stereo cameras, to determine when the user “touched” the hologram.

An additional benefit of such illusory touch-sensitive buttons is that they are actually touch-free, this protecting both the user and the hardware from situations where one can harm the other.  For instance, if your hands are dirty (such as when cooking in the kitchen), a touch will not dirty the button.  Conversely, if the hardware were dirty (such as on a shared touch interface in a public place during a pandemic), the user need not come into physical contact with the potentially dangerous hardware.

A true hologram

Unlike some current uses of the term “computer hologram” (which refer to a process of using a computer to dynamically render the correct perspective 2D image of a virtual 3D object in reaction to a user’s changing position in 3D space relative to that object), a “true hologram” is an imaging process that captures the three dimensional information of an object so that it can be reconstructed optically simply by shining light on the imaging film (no computers involved).

In this video, one of the first holograms we made for this project is illuminated by a single LED (on the stand in the background) and the reflected image (of what will later become our trackpad) can be seen to appear to float at a fixed distance from the film.  (The paper is placed near this distance to help show where that image is.)  Because they are a true hologram, these UI elements will have full stereo and motion parallax.

Making the hologram adaptive

Holograms have a very small optical bandwidth.  This means that the holographic film is effectively transparent to typical illuminations, such as light coming from a computer display or from room lighting.

In this demonstation, we place the imaged holographic film on top of a display.  Because of the small optical bandwidth, we can see through the film to the underlying display (lower half of the video) as well as the floating hologram.  Meanwhile in the upper half of the video you can see that the user is controlling the scrolling of a map application by using the holographic interface.

Because the hologram is transparent, it can also be adaptively turned on or off as needed simply by toggling the LED that illuminates it on or off.  The beginning of the following video shows the user turning on the holographic trackpad using a circular trigger gesture.

Another benefit of having the hologram on top of a display is that you can dynamically add active elements to the hologram by changing the display—for example, highlighting a holographic button to give feedback when the user ‘touches’ it.  This presents a problem, however, because the hologram and underlying display appear to the user to be on different planes. To solve this, we measure the position of the user's eyes so that we can determine which pixels of the display are aligned with the hologram from the user's vantage point.  We then draw virtual cues at that location, maximizing the visual connection between the hologram and the display.

In this video you can see the arrow buttons filling in with a highlight when they are actuated.  The parallax is calculated so that the button fill appears inside of the hologram to the user (though not necessarily to the video camera).

Interaction with a keyboard

Here the holographic film is bonded to a thin, flexible, toughened polycarbonate sheet that forms the outer layer of a force-sensitive keyboard.  The pressure-sensitive keyboard functions through the hologram, which is recorded into a very thin acrylic photopolymer.  The hologram is made off-axis so that the user doesn’t see a direct reflection of the illumination source.

In this video you can see the user typing using this hardware.
The full stereo and motion parallax of the integrated system can be seen here: as the camera’s vantage point changes, the hologram maintains its apparent position in space.

Putting it together

Here we see the system being used like a typical trackpad, with a mouse movement region and mouse click buttons.  The user’s actions are modelled on the display to show where his finger is and when it is engaged with the floating interaction surface.
Finally we see the user moving files using the Helipad.  Audio cues give the user additional feedback on the interaction.

In practice, using the hologram for mouse movement works well, but the lack of tactile feedback makes use of the buttons more difficult.  This project led to further work by the team on “active” floating displays, where the visual indicators are fully dynamic and programmable.

People