IN 1963 IVAN SUTHERLAND CHANGED THE RELATIONSHIP BETWEEN HUMAN AND COMPUTER. While an electrical engineering doctoral student at MIT, he developed Sketchpad, the first computer program to use a graphical user interface (GUI). Human-computer interaction was born. As Sutherland explained, “The Sketchpad system makes it possible for a man and a computer to converse rapidly through the medium of line drawings.…In the past, we have been writing letters to rather than conferring with our computers.”1 Sutherland used a light-sensitive pen to draw directly on the nine-inch display screen of the TX-2 computer then found at MIT’s Lincoln Laboratory. He realized that the resulting strokes could be manipulated on the screen using set rules, locked into a single image, moved around, copied to build more complex images, and even stored in a library to be used later. And all of this could happen in real time, facilitated by a general user without programming expertise. Through Sketchpad, Sutherland threw open the doors of the computing castle to engineers and designers. He made a short film of his discovery, Sketchpad: A Man-Machine Graphical Communication System, which became a cult classic in computer research circles.2 Computer-aided design (CAD) programs and object-oriented programming sprang from the ideas he introduced.

THE ULTIMATE DISPLAY

IVAN E. SUTHERLAND | 1965

We live in a physical world whose properties we have come to know well through long familiarity. We sense an involvement with this physical world that gives us the ability to predict its properties well. For example, we can predict where objects will fall, how well-known shapes look from other angles, and how much force is required to push objects against friction. We lack corresponding familiarity with the forces on charged particles, forces in nonuniform fields, the effects of nonprojective geometric transformations, and high-inertia, low-friction motion. A display connected to a digital computer gives us a chance to gain familiarity with concepts not realizable in the physical world. It is a looking glass into a mathematical wonderland.

Computer displays today cover a variety of capabilities. Some have only the fundamental ability to plot dots. Displays being sold now generally have built in line-drawing capability. An ability to draw simple curves would be useful. Some available displays are able to plot very short line segments in arbitrary directions, to form characters or more complex curves. Each of these abilities has a history and a known utility.

It is equally possible for a computer to construct a picture made up of colored areas. Knowlton’s movie language, BEFLIX, is an excellent example of how computers can produce area-filling pictures.3 No display available commercially today has the ability to present such area-filling pictures for direct human use. It is likely that new display equipment will have area-filling capability. We have much to learn about how to make good use of this new ability.

The most common direct computer input today is the typewriter keyboard. Typewriters are inexpensive, reliable, and produce easily transmitted signals. As more and more online systems are used, it is likely that many more typewriter consoles will come into use. Tomorrow’s computer user will interact with a computer through a typewriter. He ought to know how to touch type.

A variety of other manual-input devices are possible. The light pen or RAND Tablet stylus serves a very useful function in pointing to displayed items and in drawing or printing input to the computer. The possibilities for very smooth interaction with the computer through these devices is only just beginning to be exploited. RAND Corporation has in operation today a debugging tool that recognizes printed changes of register contents, and simple pointing and moving motions for format relocation. Using RAND’s techniques, you can change a digit printed on the screen by merely writing what you want on top of it. If you want to move the contents of one displayed register into another, merely point to the first and “drag” it over to the second. The facility with which such an interaction system lets its user interact with the computer is remarkable.

Knobs and joysticks of various kinds serve a useful function in adjusting the parameters of some computation going on. For example, adjustment of the viewing angle of a perspective view is conveniently handled through a three-rotation joystick. Push buttons with lights are often useful. Syllable voice input should not be ignored.

In many cases the computer program needs to know which part of a picture the man is pointing at. The two-dimensional nature of pictures makes it impossible to order the parts of a picture by neighborhood. Converting from display coordinates to find the object pointed at is, therefore, a time-consuming process. A light pen can interrupt at the time that the display circuits transfer the item being pointed at, thus automatically indicating its address and coordinates. Special circuits on the RAND Tablet or other position input device can make it serve the same function.

What the program actually needs to know is where in memory is the structure that the man is pointing to. In a display with its own memory, a light pen return tells where in the display file the thing pointed to is, but not necessarily where in main memory. Worse yet, the program really needs to know which sub part of which part the man is pointing to. No existing display equipment computes the depths of recursions that are needed. New displays with analog memories may well lose the pointing ability altogether.

OTHER TYPES OF DISPLAY

If the task of the display is to serve as a looking glass into the mathematical wonderland constructed in computer memory, it should serve as many senses as possible. So far as I know, no one seriously proposes computer displays of smell, or taste. Excellent audio displays exist, but unfortunately we have little ability to have the computer produce meaningful sounds. I want to describe for you a kinesthetic display.

The force required to move a joystick could be computer controlled, just as the actuation force on the controls of a Link Trainer are changed to give the feel of a real airplane. With such a display, a computer model of particles in an electric field could combine manual control of the position of a moving charge, replete with the sensation of forces on the charge, with visual presentation of the charge’s position.…By use of such an input/output device, we can add a force display to our sight and sound capability.

The computer can easily sense the positions of almost any of our body muscles. So far only the muscles of the hands and arms have been used for computer control. There is no reason why these should be the only ones, although our dexterity with them is so high that they are a natural choice. Our eye dexterity is very high also. Machines to sense and interpret eye motion data can and will be built. It remains to be seen if we can use a language of glances to control a computer. An interesting experiment will be to make the display presentation depend on where we look.

For instance, imagine a triangle so built that whichever corner of it you look at becomes rounded. What would such a triangle look like? Such experiments will lead not only to new methods of controlling machines, but also to interesting understandings of the mechanisms of vision.

IVAN SUTHERLAND
“Sketchpad: A Man-Machine Graphical Communication System” 1963

There is no reason why the objects displayed by a computer have to follow the ordinary rules of physical reality with which we are familiar. The kinesthetic display might be used to simulate the motions of a negative mass. The user of one of today’s visual displays can easily make solid objects transparent—he can “see through matter!” Concepts that never before had any visual representation can be shown, for example the “constraints” in Sketchpad.4 By working with such displays of mathematical phenomena we can learn to know them as well as we know our own natural world. Such knowledge is the major promise of computer displays.

The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.

1. Ivan E. Sutherland, “Sketchpad: A Man-Machine Graphical Communication System,” AFIPS Conference Proceedings 23 (1963): 8.

2 A digital copy of the 16mm film can be found on YouTube.

3 K. C. Knowlton, “A Computer Technique for Producing Animated Movies,” in Proceedings of the Spring Joint Computer Conference (Washington, DC: Spartan, 1964).

4 Sutherland, “Sketchpad: A Man-Machine Graphical Communication System,” in Proceedings of the Spring Joint Computer Conference (Washington, DC: Spartan, 1964).