The Trackpoint is a little device with a big story. In this case, a story of the down-and-dirty, roll-up-your-sleeves, be persistent, aspect of innovation. It is also a story that I would rather you hear from the person who told it to me, Ted Selker, the person mainly behind the technology. So here it is, in his own words:
Bill - let me tell you a little story about Trackpoint!
It all started in 1984 when I got my hands on Stewart Card and Tom Moran's new book, Human Computer Interaction.
I was amazed to notice that in the first reported mouse experiments by Bill English and Doug Englebart in the first few minutes people did better with a simple knee bar than with a mouse…. I thought, now a knee has no big representation on the pre- central gyrus of the central sulcus for perception or on the post-central gyrus for motion like the hand does….. So, why would it start out doing better? A few pages later I read some data that showed that going to the mouse took more than .7 seconds and coming back to the keyboard from it took more than .9 seconds…... Ahha! , I said- if a hand doesn't yet know how to use a mouse it won't yet be better at pointing than if a knee that doesn't yet know how to point … and so the time of transit to and from the keyboard will dominate.
I went about thinking of places to put a pointing device that the hand could nab from the home row position… I tried in front of the keyboard … and yes made a mechanical claptrap that year that had X and Y excursion to do this. I also played with an erasure on top of a X/Y joystick that the thumb would control below the space bar. I considered a joystick between the hands, but then started working on using the J key as a joystick.
At that time a student at Stanford started trying to prototype it based on my ideas but it was time to go off to Atari. I thought about it there at Atari, but nothing ever got out of the Atari Sunnyvale research lab….
I wrote it up as a patent in April of 1984 or so and submitted it. (I can get more data to you about this if you like) I believe I happened to be talking about this idea at CHI around the Tektronix people… which begat J-mouse … years later after they left Tektronix.
I went to PARC for a year… and did many fun things there with automatic graphic design to a narrow-waisted mouse that can be held like a pen…. I went to IBM In August of 1985 and I did tell them I had this patent pending. (I also had a stylus pointing device patent that should be more exploited .. but that is another story too). So I didn't work on it till a math department colleague Joe Rutledge who wanted to break out of the math world chose it as the idea I had he wanted to collaborate on. We worked on it on the side to begin with… in 1987.
Joe chose it instead of my big visual interface or intelligent user interface ideas because it seemed that it could change a current product in a noticeable way easily…. I thought of it as a science fair project that would take 6 weeks… I was wrong.
I fixed a microscope that was being thrown out by reglueing its mirrors; I found an oscilloscope that wasn't being used and then found an on piece of bench in the back of a lab to use. At first my idea was to use conductive foam beneath a key as the joystick. I put copper foil corners on the top and a foil bottom… I found that the conductive foam at Radio Shack didn't have the hysteresis that the other chip-holding foam had. The sensor was great in that it was sensitive and fit under a key and was easy to make- but it sucked in that it was where I learned about the problems with force-sensing resistors. They seem to always be nonmonotonic and non-repeating … at more than 10%. This made calibration impossible. Next we tried making a cute cantilevered under-the-key system for making a 4-piezoelectric sensor . These were beautiful - sensitive and clean but they needed to be grounded out to have a stable beginning to their excursion. We made a ground that the piezoelectric devices would touch whe n the key was not pressed….Ugh… piezioelectrics have drift when pointing for long, and grounds are not reliable at the tiny signal level they have. Piezioelectrics are not at all good for DC analog measurement. I got so mad that I went to Ashok Chandra, my boss, and said I wanted him to spend $500 on a Measurement Systems joystick; one needs a good signal to do anything! He looked at me gravely, let the moment stretch and then said, "Get two".
Joe had programmed on the Eniac in the early 1950's and built many things in his life. I now easily taught a brilliant Joe with a photographic memory digital electronics. And in a few weeks we had made a circuit that put out mouse quadrature signals.
Joe learned Basic easily. We did the really important piece; we built a human factors testing maze that ran on a discarded PC. The maze was of 16 characters all over the screen, some short distances, some long, to see how the joystick worked with Fitt's law.….
Oops! I had thought that placing the joystick in the keyboard was the hard part - making it point well was harder. Hmmm, somehow it was hard to control; we fiddled with feedback resistors and capacitors trying to create transfer functions to make good pointing… yuk…
So now we decided to tear off the signal conditioning stuff on the prototype boards and go into an A/D board in the PC. This was the next cool thing.
We tried various transfer functions: linear, S curve, but found it confusing. So now we built a graphical interface for designing a transfer function that went through arbitrarily drawn places in the force-to-velocity space. Just like others' joystick experiences, it skated over the material but when you wanted to stop you couldn't. It would overshoot and you had to move it slowly back to where you meant to stop. Why did you have to go past something you were selecting, stop and backtrack? We read about overshoot in joysticks. People had theories that it was the fact that rate control was the derivative of position that made position control superior. We fiddled and fiddled - after a short time though we discovered a weird thing. We made a transfer function for the pointing stick that hurt your finger but could make selections 25% faster than any reported joystick ever. Grrr- it hurt and there was a dent in my finger when I used it. Why was it faster when I pressed with 5 kg of force (thank you Measurement Systems for a sturdy sensitive sensor)? Now Joe and I were having fun. Gosh, we even got a 6 foot wide office to turn into our lab.
At first I thought that this magnificent improvement in selection speed was due to the increased dynamic range of muscles for force control. It felt so much like two actions, a coarse motion and a fine motion, that I even looked to no avail for literature about different kinds of muscular control. We made a program to test finger force control. The idea was a circle on the screen representing X and Y force space that you had to hold a cursor within. The force space made the cursor feel like it was on spring so you had to press a certain hardness to get it to the circle. The circle size represented your stability and its position on the screen represented how much pressure you had to apply. Wow was this weird… we discovered you could only hold the finger in the circle with like 5 or so bits of resolution. Gosh, so little control. And wow, the force control didn't get much better beyond 8 oz of pressure. We drafted a paper, and it got rejected from CHI… sigh… We were too busy to deal with it. So we hummed and hawed and decided that the pain stopped you from pressing the cursor to go faster than you could control it. Eureka, by making a transfer function that didn't go fast we could make selections 25% faster than the literature reported with a light touch that didn't hurt! We called this the eye tracking speed, though we never took the time to get eye tracking equipment for this project to measure a person's ballistic eye motions. We were happy, so we tried others using it. Yes, users were faster at making selections, but they didn't like the sluggishness. So we added a steep "turbo charge" gain segment to the transfer function where you press hard to make the cursor goes super-fast. This didn't change the data but made the users happy. We didn't know why.
Many months later, when we were using it in Window applications (on my lovely 3645 Lisp machine) I finally realized that the "turbo charge" part allowed me to toss the cursor across the screen …to change window context. Fine motion was also a concern, and we went back to our 5 bit force resolution result and thought that if you don't have much control at slow speeds then tremor would increase inaccuracy and maybe a constant speed would be easier to stop on a dime with…. Sure enough a constant speed selection gave a 15% improvement over controllable slow speed in activities like single pixel selection.
Before any of this worked, somehow the inventor of RISC, John Cock came to hear about our input device. He liked it- but then, a few weeks later he had a stroke. Abe Peled , VP of IBM computer science research, went to see him. When asked what he wanted, John surprised the visitor by stating "I want one of Ted's pointing devices so I can use my computer in bed." Luckily it was now working well enough for a good demo. James McGroddy (soon to be General Manager of research) came to see what we had done. Sadly, the pointing device was not packaged to send to the hospital and work on someone's computer yet. But now people were interested. Ashok ordered us to be extravagant and make John Cock and 96 other interesting people hand-made pointing stick adorned keyboards . At this time, IBM made me abandon my patent as terms of continued work on my device and on my research and thesis about adaptive help… it was a difficult moment and an exciting one.
We wanted to get the PCcompany to see our in-keyboard pointing device. To get it to really work in a keyboard was complicated. We had to replace the large Measurement System's joystick package. We bought the silicon strain gauges used in the fancy joysticks but it took us a month to learn how to mount them. Picking up one of these expensive silicon gauges by its gold leads broke it. I borrowed an X/Y/Z stage from the microscopy guys. It was so difficult to manipulate the tiny thing under a microscope. Finally I had the idea of pressing a piece of clay into the sensor holder to pick it up. At that point the clay mounted to the desk and pressing it close to the sensor moved it towards a strain beam it could be mounted on. Pressing the lump near the desk moved it very very little.,. When the strain gauge was in place, the superglue came out. Others at IBM research started using our micromanipulation technique.
The PC division had us show the device to the people making a laptop; they couldn't be convinced it was necessary or would be reliable. We showed it to a bank- the bank wrote a letter stating that using this would save them on the real-estate needed in an office to leave mouse pads at the clerks' counters- they would buy $40,000,000 worth of IBM PCs if the company would sell them PCs with the pointing stick in it. The PC company declined. I showed the pointing stick to the design team in Armonk- one Richard Saper gave me 1 minute, he tried to draw a circle - we had optimized for text editing - he said it can't draw a circle and walked out… We learned to stop optimizing for text editing and Saper became an important supporter and even chose the color red for what would be called the TrackPoint. We always hoped that a driver would make the cursors optimize for the purposes they needed the cursor for- this hasn't happened yet.
We finally got so frustrated that I told the PC division that if they didn't use it I would publish in 6 months. I reminded them of my "bet my badge" deadline as the time approached. At the last moment, the lawyer relented and helped me submit the paperwork correctly
As soon as we showed the device in Cambridge, England at Interact it was obvious it was a great hit.
The crowd was relentless and enthusiastic. One Jim Lewis from IBM Boca Raton was there. He went home and told his boss Bob Lawton that they should consider this device for the L40SX computer. We were invited to PC land again. This time we were told that it would go into the product. Jim Lewis ran excellent experiments. The Entry Systems Technology group was responsible for testing out new stuff.
They had a code name for the project, "Castaway", and that's what they tried to do. To bolster confidence in our judgment, human factors experiments were run in Austin, Boca, Lexington and Yamato. It was a circus. I remember the way the PC division was- I would set up meetings with everyone, they would be 1 mile from each other in a pentagon-shaped building, we would run from Testani's to Condon's office, to Lawton's to Lewis' … no one joined each other's meetings so they could say different things and I would try to work on the constellation of comments they made. The people in Austin, Lexington and Boca seemed disorganized around our innovation. As soon as the product was accepted by Toshiuki Ikeda of Yamato team, they asked me "Do we have to pay this ½ million to have EST do all these things?" It could have cost the product - I looked at the list of things that had to be done to productize Castaway- only I and Joe and Bob Olyha knew how to do them and most were actually already completed….
"Please work with us," I said. I was vocal and in the end I may have been the reason that the people causing friction for new technology in EST were disbanded the following year.
The process of getting the pointing device into the Yamato project was the end of a lot of stories I won't enumerate here … more difficult demonstrations and meetings and pressure from the top IBM brass, more help from the bottom, more obstacles everywhere. The distinguishing feature, however was the team approach to decisions and execution throughout the IBM Japan culture, a direct contrast to the processes we had encountered in the US IBM PC groups in the early 1990s.
We went to Yamato, the first time I had been in Japan. Ashok Chandra sat me down to help me integrate the data from seemly inconsistent experiments done at various labs in the world. The night before the presentation, we fit it together in its final form: the reason people had conflicting data was that the Yorktown experiments tested hours and weeks of use, Boca Raton measured 15 minutes of use and the Yamato data measured the first minute of use of the new device. Yamada-san stood up and gave a 45 minute talk about one cell on the learning curve matrix, the only one that showed the pointing stick being worse than a trackpad or trackball,for the first 15 selections. I got up and pointed out to everyone's amazement that while the Japanese's data was the most carefully taken and cleanest ever produced, it all represented a user's first experience only: this was news to the Japan team. Yamada had been working with Alps on a trackball that he was fond of. Still, Yamada agreed with me to everyone's am azement. Our work showing how the different experiments corresponded to different parts of a learning curve was the hit that mattered. But then everyone agreed that the first few seconds of use might be the most important part of selling a product. Unlike the American decision process, everyone in that meeting went to the next level up meeting and we were required (in an embarrassing show) to repeat our conflicting talks again. This happened to an even larger crowd by afternoon including everyone that had been in the other two meetings. It included everyone agreeing to what had been said, allowing consensus. Yamada had been encouraging us to introduce motion in our isometric joystick. We had believed there was no space between keys for such motion. We built one for him anyway that bent and brought it to Japan. We had experimented with many shaped tops; we had convinced ourselves that a cup-shaped top (much like one that comes with a ThinkPad today but much smaller) would hold the finger with its edges . Product manager and man with deep insight Ikeda and top engineer Arisama Naito both hated the way it felt- Ikeada pushed down hard and showed me how it left a dent in his finger. We had one week to fix the first experience problem and the feeling. The fourth level manager Kasuaga-san stared me in the eye after we met with Mito the general manager of IBM Asia Pacific and got the go-ahead… he said now we put it in the product you must make it work. It was one of the scariest moments ever.
We included Yamada as tour guide and judge. We needed his interest and buy in to at least not be against anything besides the trackball. He pushed us to do better work than we thought we could. We went to Toyku Hands where I bought styrofoam, sandpaper, cork, sorbothane, rubber, etc. We went to a ping pong shop- we went everywhere with me and my rechargeable dremel making Trackpoint tops… After several days and many dozen tops, we had a top that was gooey enough to not slip, gooshed enough to hold the finger, and bent enough to give a novice proprioceptive feedback,,,,, that improved the first 15 selections by at least 15%, enough to match the pointing speed of a trackpad or trackball.
This I hate to say was the beginning of a 9 month long list of tiny successes. Yamato assigned a Michio Suzuki full time to report to us the bugs he found with programs … and he found problems daily for months. Bob Olyha worked on circuitry- improving everything about stability and calibration and many other things.
The rubber I specified, a 55 durometer post with a 145 mm top made of 1.5mm gushy stalks and a 1 mm low durometer latex top , did not make the IBM Rubber Center of Competence happy. They said it wasn't FDA approved… my father pointed out that condoms are made of latex …. He had worked at the Bureau of Standards on rubber design during "the War"- I hired him to eliminate the plastic-filled-with-rubber top that would have killed the product; it was over Ikeda's dead body that I spent $20k on a tool to make my father's beautiful rubber top- the Portland company got screwed as we used the Japanese partner of IBM to make the tops…the tops were not ready for production when the product shipped, I was adamant that the new tops were necessary… Within a week all the plastic tops in the field began breaking… they got slippery and then would require many times an ergonomically reasonable pressure to function… My father's tops were ready as the problem emerged in customers' hands. No one cared that they were expens ive. While my father knew that the tops were going into product, he never got to see it as he died that week.
I challenged myself to cost reduce the device by one third each year and improve the product. At first it was special needs buttons that were more comfortable and locked, eliminating the long small signal kapton wires, integrating the post with the circuit board, inventing a new kind of A/D to eliminate a high noise chip, inventing more grippy tops that wore better and cost less, inventing a 3-D sensing post, incorporating invention of "negative " inertia that I worked with Rob Barrett to create, integrating work from multiple pointing stick experiments, adding and press to select- all made it better. A mouse communication language allowed more control from the driver. Better manufacturing processes and jigs simplified the product too. An inplane joystick invention made the post more robust in several ways. An integrated mechanical fuse that I worked with Barton Smith and Todd Wyant to create that reset automatically was fun to design too. Each of these features is a story in iteself. Steve Ih de put numerous improvements into the drivers all the time. All the sticks ‘til recently shipped with an algorithm that made it so a collaborator using another pointing device plugged into the keyboard would have conversational (1/3 second) pause needed to take the pointer away from the current driver of the UI. This was hugely helpful for collaboration. We shipped a teaching screen saver on OS/2 that showed how to use it and how to use the buttons.
The ones that got away are more poignant. I designed adaptive algorithms we were exploring- we were able to raise the tracking plateau by 50% for some people, and if we had had the go ahead we could have made that stable and increased performance tremendously. It seems some people used tendon flex to improve pointing- we found this looked like overshoot and we had trouble making the adaptive algorithms stable in the time box we gave it. We made a special application that was a game people played to improve their pointing –as their game play improved the transfer function improved too! We made a surgical tool that used a Trackpoint to allow tremor-free use of a camera from a laparoscope. We made a selector for the FAA to do ground traffic control that saved a multihundred million dollar contract for IBM with the government. I would have loved the product to change cursor movement approach for form filling, text editing and graphical applications; again that could probably double performance. Yes, I s till want to do it NOW. We built a gesture language into the Trackpoint that can be accessed in the firmware; the only aspect the driver exposes is Press to Magnify or Press to Select. We created probably two dozen haptic feedback Trackpoint designs. They improved novice performance and were loved by the special needs community. I did a preliminary study that showed how novices selected faster with it; the product group saw no need to spend the money for that. We made a science experimental test bed to teach physics that never shipped; we made many versions of multi-Trackpoint keyboards that never shipped. We made many other things too- a versatile pointing device for the table called Russian tea mouse allowed for full hand, thumb, finger, or in-the-palm use. We made pen like stalks that allowed selection without taking hands off the keyboard. We made devices that used one set of sensors to run two input devices. We made an electormechanical design used by one special user. We found that brushing th e top instead of pressing it could give amazing dynamic range, at the expense of having to cycle the finger for long selections. The joystick for this had no deadband, it had an exquisite sensitivity and control … we never made an inkeyboard device that shipped with this alternative set of algorithms and scenario. I designed better grippy top ideas that never made it; also better sensitivity solutions that never made it too. And I hate to say it but there are many other improvements that I made or would like to make that I could elaborate further on but will stop here…..
