(opens in new tab)
ActiveInk aims to capture the freedom of a traditional pen, making consuming information and externalizing thoughts about it a cohesive digital experience. The system’s active ink turns pen strokes into a means of executing an intended action, such as highlighting, hiding, or cutting specific information.Whether at work or at home, people are regularly gathering and interpreting information to build their base of knowledge and gain a deeper understanding of the world around them. In this endeavor, they encounter data in many shapes and forms—lines of text in books, timelines and charts in magazines and newspapers, photos in print and online. They’re searching webpages and browsing interactive maps to identify good schools for their children or to find a great house to purchase; they’re monitoring their medical and financial data to make informed decisions in their life.
When making sense of information on paper, many people engage in active reading, crossing out parts of data visualizations that are irrelevant to their current analysis, identifying outliers in a scatterplot, or marking passages of interest. They may also jot down their hypotheses and interpretations directly on top of the data, such as drawing a correlation line. As people conduct more of their lives electronically, it’s natural to offer them a way to engage in these active reading activities on their computers with a digital pen.
Microsoft research podcast
Collaborators: Silica in space with Richard Black and Dexter Greene
College freshman Dexter Greene and Microsoft research manager Richard Black discuss how technology that stores data in glass is supporting students as they expand earlier efforts to communicate what it means to be human to extraterrestrials.
But it’s not enough to try to replicate the pen-and-paper experience. In fact, it might not be possible, as even with hardware and software advances in the past decade, writing on a piece of glass doesn’t feel the same as writing on paper. The digital experience doesn’t match the sensory aspect—the tactile, visual, and auditory feedback—of the analog experience. Affordability and reliability are also among the many reasons why people might choose pen and paper over the digital option (opens in new tab). Instead of seeking to emulate the analog experience, our aspiration is to uncover the unique capabilities of the digital pen. Our research aims to give the digital pen superpowers for marking documents in ways that go beyond what is possible with physical pen and paper.
The power of active ink
With the web-based system ActiveInk (opens in new tab), we’re enabling users to seamlessly transition between exploring data on screen and externalizing their thoughts on screen using pen and touch. ActiveInk, inspired by such previous work as InkSeine (opens in new tab), allows the natural use of pen for active reading behaviors while supporting analytic actions on underlying data by activating any of the ink strokes laid on the screen. So marks serve not only as reminders of an action to be applied later—whether that be extract, remove, enhance, or ignore that particular information—they become the means to accomplish the action. It’s active ink. More than a set of strokes, it facilitates an interaction between users’ thoughts and the underlying content.
Let’s say, for example, you’re working with a map of the United States to identify those regions of the country where you might want to live and work. You decide to focus on only the East and West coasts. You can cross out or scribble across the Midwestern states—your indication that the information is unnecessary—and use the same markup to initiate the desired action, which in this case is to delete those states.
With one pen, users can mark things freely in a variety of colors and stroke styles and execute or fix them later. Activating these annotations by tapping on them reveals a set of commands, such as highlight, remove, and label, and analytical functions, such as compute a regression. The digital pen can also be used as a precision input to cut off outliers in cluttered visual.
The driving force
Switching between digesting data on screen and externalizing resulting thoughts and analysis with pen and paper has physical and cognitive costs, including time expended writing down enough notes to remember or retrieve later the chart or table you need, for example. At the heart of ActiveInk, which we’re presenting at the ACM CHI Conference on Human Factors in Computing Systems (CHI) (opens in new tab), is three principles to minimize those costs:
- Provide space to think. Sensemaking is not linear: People need to cross-reference different pieces of information from different files. They may need to go back to previously seen charts, look for new ones, or skim through a long text to identify a missing piece of information. Thinking often requires capturing fleeting thoughts and revisiting them multiple times before finalizing an analysis. ActiveInk provides users with an infinite canvas to do that. They have free reign to drop information in and reorganize it at will. Marks made to an individual piece of data stay with it as users move around to available parts of the canvas, never running out of room to work.
- Empower the digital pen to both interact with data and allow for externalizing thoughts. Two interface strategies were developed and evaluated to operate on the infinite canvas:
- In the prefix method, users select an action to be taken from a menu of options either with a finger on their non-dominant hand or the pen in their dominant hand, and the selected action is applied as the pen is used.
- In the postfix method, everything done with the digital pen results in a pen stroke that can be activated after the fact, as described above; the marks that result in the action remain as a means to recall or undo the action.
- Avoid requiring memorization of gestures. The ways people annotate information is different, so instead of asking users to change the way they work, ActiveInk supports how they naturally use a pen.
The best of both worlds
For the prefix method of interacting and externalizing to be worthwhile, it requires the user to have some idea of what they want to do with the information, which isn’t always the case when interpreting data in the moment. The postfix method addresses that and encourages free thinking, as it isn’t necessary to choose the tool before making a mark. Whereas the prefix interface is similar to what users have seen in other programs where editing tool options are visible throughout the experience, the postfix interface requires a tapping on the ink to reveal the action options, a dragging motion to preview it, and finally a lifting of the finger to execute it. In a qualitative study involving eight participants, we found that the prefix and postfix methods were leveraged by users in different scenarios: the former for successive actions and the latter for more note-taking. To offer the best of both worlds, we implemented a hybrid of the two methods.
ActiveInk has the potential to make the digital pen a power tool for both interacting and thinking with information, providing a single space for people to compare and contrast files from different sources using such functions as those offered by Excel and Power BI, as well as note-taking tools. ActiveInk enables people to (th)ink with data, realizing the thoughts they have in ways that are not possible with pen and paper.
Human-computer interaction