CHAPTER 16

The Pariahs

Dick Shoup had it down to a routine. He would pedal his bicycle up the tree-lined hill to the front of Building 34, lever open the door of the main entrance with his foot and pedal through without disembarking, then proceed straight down a narrow hallway, the tires of his bike whishing softly on the worn carpeting like wind through a sparse wood. Finally he would roll to a stop next to the graphics lab.

If only his work inside that room could proceed with as few impediments. But no: Dick Shoup had invented a technology that would stand the science of video on its ear, and he was close to getting fired for it.

The machine was called Superpaint. It deserves a place in history as the only invention too farsighted even for PARC’s Computer Science Lab. And all because it thought in color.

The notion that an excess of ambition could make a talented inventor into a pariah at CSL sounds preposterous on its face—more like something that would happen to a Gary Starkweather in a place like Webster, and the very antithesis of what should have happened at PARC. Yet every organization of human beings eventually comes to cherish its own orthodoxies, and PARC was no different. When a group pursues a goal with the single-minded tenacity the Computer Science Lab possessed under Bob Taylor, the potential for intolerance is even greater.

“It was hard to be a renegade in that lab,” Shoup said years later with a regretful sigh. “You could be a maverick, but only a maverick of a certain kind. And I guess I was just the wrong kind.”

 

He had not started as an outsider. Quite the contrary. One could scarcely imagine a more lace-curtain computer science pedigree than Dick Shoup’s: Ph.D. from Carnegie-Mellon and employment after graduation at Berkeley Computer Corporation, followed by selection by Bob Taylor as one of the elite six to join PARC upon BCC’s demise. In the small society of the Computer Science Lab, this was the closest thing to coming over on the Mayflower.

You would have to know Shoup very well before discerning the heretic’s soul underlying those sterling academic bloodlines. One clue was his interest in things that struck even some of PARC’s free thinkers as a little outré, like his Transcendental Meditation group, which gathered every morning in a PARC commons room to do its thing to the voice of the Maharishi Mahesh Yogi on mail order tapes.

“Dick was a little different from everybody else,” related his friend Alvy Ray Smith. “He’s sort of a crusty guy, and he’s not political, and he’s very stubborn, and I think this is why he’s as good as he is.”

Had he known how stubborn, Bob Taylor might not have given Dick Shoup such a long leash at the very outset of his PARC career. Shortly after Shoup got to CSL Taylor welcomed him into his office to discuss what he wanted to do. Shoup did not have the clear-eyed convictions of his colleagues Lampson, Deutsch, and Thacker. Instead he saw so many fascinating paths laid out before him that he was stymied by the need to pick only one. Finally Taylor drawled: “Why don’t you take a year to figure out what you want to do?”

With Taylor this offer was never as open-ended as it might seem at first glance. He figured that great computer scientists left to their own devices (and subtly guided by the Impresario’s hand) would invariably find their way to the grail of interactive distributed computing. But Shoup took his new boss at his word. Within a month or so he had decided to pursue a course of research in video computer graphics.

On the surface his choice sat squarely within the mainstream of Taylor’s vision. After all, Taylor had underwritten through ARPA the first computer graphics “center of excellence” at Utah and hired two of its graduates, Bob Flegal and Jim Curry, as his very first recruits to CSL. Nor was it entirely random on Shoup’s part. His interest in video went all the way back to his high school days in western Pennsylvania, when he spent weekends repairing TV sets for his small-town neighbors.

Adapting video raster displays to interactive computers, he knew, raised a host of intriguing technical issues crying out for further study, especially if one desired to do interesting things with the image. One would need a way to store a whole frame’s worth of digital data in memory at a time, for example. The technology to do so was known as the frame buffer. What Shoup proposed was nothing less than the biggest and most flexible frame buffer anyone had ever seen.

 

Simply defined, a frame buffer is a box holding a hell of a lot of memory. More precisely, it is a grid in which the memory is arrayed to correspond with a video frame, so that one or more bits of memory account for every “pixel,” or “picture element,” on the display. Activate the bits of a frame buffer in any given pattern and the exact same pattern should appear on the screen. Connect the buffer to a computer and you can rearrange those same bits—and the corresponding image—according to any algorithm you can devise, the way you might rearrange colored marbles in a partitioned box.

With the assistance of Flegal, Curry, and a French graphics expert named Patrick Beaudelaire, Shoup spent more than a year devising and assembling his memory giant. The first prototype of Superpaint went operational in Building 34 on April 10, 1973, just a few days after the Alto, which was being built in a basement room directly under his ground-floor video lab.

While Cookie Monster was marching across the Alto screen to the delighted gasps of the PARC faithful and their visitors, Dick Shoup was seated alone before a black-and-white video camera, holding up an index card on which he had scrawled, “It works, sort of.” The system recorded the image of his face and the card in buffer memory in accurate detail—save for the bright red-orange of his droopy mustache and collar-length hair—and stored it on a conventional computer disk as a pattern of bits. (“It survives to this day,” he said in 1998.)

Within a few months he had added a kaleidoscopic variety of video inputs, including live television, videotape, and videodisc, as well as hardware and software to allow him to alter the images he grabbed from the screen. The finished product was the first fully video-compatible frame buffer ever built. It was also a vector apart from the computer his colleagues had assembled in the basement. Where the Alto fit under a desk, Superpaint occupied two cabinets, each standing five feet tall and holding thirty-three memory cards. Its nearly two and a half million memory bits (in semiconductor chips worth about $100,000) meant that each pixel in a video frame with a resolution of 486 by 640 pixels could be addressed by eight bits. The system required two separate display monitors, one to show the image to be manipulated and the second a menu of electronic “paintbrushes” with which it could be altered in color or pattern. “No question about it, this was a big chunk of hardware,” Shoup recalled fondly.

Superpaint was a uniquely agile and adaptable graphical tool. One could “grab” a frame from a videotape, disc, or directly off a television screen and manhandle it by changing its colors, flipping or reversing the image, bleeding it across the screen, even animating it. The key was the ratio of eight bits per pixel, which allowed the user to tune every dot to any one of 256 color values. You could freeze a random frame from a taped episode of, say, Star Trek, overlay it on the buffer as if you were tracing a line drawing on a blank canvas, and recolor Spock’s hair green by assigning new values to the appropriate pixels.

Yet Shoup’s fascination with color and video drew him away from what Taylor viewed as the Computer Science Lab’s principal mandate. Shoup had participated in the MAXC project like any obedient member of the CSL team; but by the time the rest of the lab shifted its attention to the Alto he had withdrawn into his personal world. Taylor was distinctly displeased at the course things were taking.

“Bob felt the whole lab needed to be working in one direction,” Shoup recalled. It was not simply that he was working on his own; more critically, it was felt that the basic premise of Superpaint would never fit in with CSL’s goal to build the “office of the future.” It was one thing to study how digital bits could be manipulated to create an image—the direction CSL had taken since the day they mapped Cookie Monster’s face to the Alto screen. To his colleagues Shoup was working backwards, starting with video images and reducing them to their digital components: What could an office system ever do with that?

“Everyone on that side of the house was interested in documents,” Shoup recalled. “Documents are pretty much black marks on white paper. Color meant TV, and that was some other world.”

The tension between Shoup and the rest of the lab intensified through the end of the year and into early 1974, as pressure mounted on CSL scientists to focus their efforts almost entirely on Alto-related projects. It seemed that the only thing keeping the simmering disagreement from turning into a full-scale break was the absence of a catalyst. Then, as if on schedule, a man arrived at PARC who really did seem to come from some other world.

Alvy Ray Smith was the quintessential 1970s dropout. A native of New Mexico, he had been a New York University computer professor until abandoning his promising academic career to drive a white Ford Torino cross-country in pursuit of the muse of abstract art. He and Shoup had first encountered each other several years earlier when Smith, an expert in the arcane mathematics of massively parallel computers, was putting together a conference panel on modular computers and someone recommended he contact Shoup, whose dissertation at CMU had covered the same territory.

They embarked on a lifelong friendship, based in part on their shared fascination with the unconventional. “Dick was always willing to talk about all kinds of other things than science,” Smith recalled. “Music, art, parapsychology, out-on-the-edge stuff.” When Smith abruptly quit his professorship in 1974 and headed for California to paint, it seemed only natural that he would surface at the home of his old friend Dick, looking for a place to spend the night.

PARC then was at a peak of creative ferment. Every day some new feat of engineering appeared, virtually demanding to be shown off to anyone with a free moment. And here was Alvy Ray Smith, curious as a cat, at large with time to spare. Shoup fairly tingled with anticipation as he drove to the research center the next morning. Seated next to him was the one man he knew possessed the temperament to “get” Superpaint. Sure enough, the machine hit Smith like a lightning bolt between the eyes.

“He came in the door and got completely entranced,” Shoup remembered. “He just deep-ended right into it.” For the next several days and nights the bewitched artist scarcely left the lab for more than an hour or two at a time. “I realized this was what I had come to California for,” Smith recalled. “You could just see it was the future.”

The time-honored technique of daubing paint on canvas suddenly seemed hopelessly antiquated. Smith’s new obsession was to get his hands on Shoup’s machine and never let go. Shoup favored the idea, figuring he needed someone like Smith on the premises to make up for his own lack of artistic skill (“I was a visual thinker, but never much of a visual artist,” he said). Attempting to secure Smith a place on PARC’s permanent staff, Shoup argued that Smith’s artistic talent and solid scientific credentials uniquely qualified him to help develop Superpaint’s full potential, like a test pilot pushing a new fighter plane to the edge of the envelope.

Among the higher-ups with their hands on the budget, this was a no sale. No one had thought to provide in PARC’s head count for an artist in residence, much less a rootless hippie like Alvy Ray Smith. Still, one thing you could say about PARC was that its rank and file was infinitely resourceful at finding ways to stretch the rules. After the personnel office refused to hire Smith as a temp or a contractor, Alan Kay came up with the idea of getting him into the building virtually as a piece of furniture—executing a purchase order for his services for a couple of thousand dollars. “I didn’t care how they did it,” Smith said. “I didn’t want a title or salary or anything. I just wanted access to the equipment.”

In no time he became a fixture in Building 34. If Dick Shoup was a maverick who blended in, Alvy Ray Smith was one who was hard to miss. Big and broad-shouldered, given to loud shirts, with a luxuriant mane of jet-black hair and a flowing hippie beard, he proclaimed the genius of Superpaint in a booming voice to anyone who was willing to listen and many who were not.

Taylor, ominously, viewed him skeptically from the start. Perhaps it was his claim to superior farsightedness, which Taylor took as a personal affront. Or perhaps the reason was that Smith seemed to have a singular talent for pressing his buttons, as he demonstrated on his very first day at PARC.

Smith was in the video lab, tinkering with Superpaint, when Taylor came up behind him, evidently intent on making sure the newcomer understood that this machine was considered to be out of the mainstream. He watched silently as Smith laboriously tuned the color settings, then asked, “Don’t you find this too hard to use?”

Smith wheeled on him, shocked at the very idea. One might just as well ask a painter if he found it too hard to wield his brush. “No, I don’t find it too hard,” was his impatient rejoinder. “Don’t you get it? This machine is revolutionary!”

Taylor walked off with a grunt, unhappy at being lectured in his own lab about what was and was not revolutionary. He had never before been reproached as a reactionary, and it stuck in his craw.

Nor was the significance of the exchange lost on Smith. “From that day on,” he said, “I realized my friend Dick was in an unfriendly environment.”

Taylor and Smith, of course, had been speaking at cross-purposes. The Computer Science Lab was a collection of engineers who weighed everything pitilessly against the question: How will this get us closer to our goal? They had committed themselves to developing Xerox’s office of the future, and anything that diverted their attention or served an alternative goal had to be discarded or obliterated. To them the glorification of fluke and luck so cherished by the creative artist seemed intolerably wasteful of time and effort. Even Alan Kay tested their patience with his penchant for drifting haphazardly through Ideaspace; but compared to Alvy Ray Smith, Alan Kay was as sober as a Presbyterian elder.

Smith soon rewarded Shoup’s faith that someone rooted in both art and science would make a powerful contribution to Superpaint’s evolution. His most important refinement had to do with the way users adjusted colors on the screen. As an engineer, Shoup had built a system with elements only another engineer could love: The controls were a set of “sliding levers” that could adjust only an image’s red, green, and blue values. The process lacked a certain necessary delicacy, almost like forcing composers to do without sharps or flats.

“Artists don’t think that way,” Smith informed him.

As he explained later, “Dick could get any color he wanted, but he had to think in terms of how you might get pink out of red, green, and blue.” (How tricky this is can be imagined by anyone who has tried to adjust the flesh tones on an old color TV using only the three primary color dials.) Making the system intuitively useful for artists, Smith recognized, required an additional set of controls. “If I gave you controls for hue, lightness, and darkness, you would know you could take red and make it lighter: That’s pink.” He called the new categories “hue, saturation, and value” and labeled the system the “HSV transform.” (Smith’s additional categories, or similar ones, survive in most video animation systems to this day.)

But Smith’s devotion to Superpaint hastened Shoup’s estrangement from his CSL comrades by making his machine seem little more than a toy for longhairs. Smith monopolized the device for hours at a stretch, twisting video images into intricate abstract forms. He would take a color test pattern and step it through a programmed sequence of the 256 color values so it resembled the skin of a chameleon placed against a kaleidoscopic background, or bleed the pattern across the screen in a psychedelic wash. “I took a girlfriend’s face and did some tricks with it, halved it down the middle and reflected it and halved that again so it was a four-way reflected face that’s hardly recognizable anymore, but still has something organic about it.”

Today such manipulation is commonplace to the point of triteness, the stuff of TV special effects. But in 1974 no one had ever seen anything like it. Soon Smith was inviting friends from San Francisco’s creative demimonde down for demos that turned into all-night Superpaint “jam sessions.” One graphics artist, an Iowan named Fritz Fisher, had been invited out to PARC by Shoup and Smith to give a talk about his work. He took one look at Superpaint and returned home only long enough to pack his bags. Back in Palo Alto he enrolled at Stanford and got a job as night watchman at CSL. For the next few years he would attend class in the daytime and tend the lab all night. “We’d come in the next morning and there’d be these elaborate designs on the machine,” Shoup said, “and we’d know Fritz had been at it.”

Among PARC scientists, however, the reaction was much less fervent, except among a handful of empathetic staff members who joined Smith in the wee hours, some displaying the furtive signs of experimental drug use. (“That was one of the dividing lines,” Smith later remarked jocularly: “You’d just look at people and know if they were dopers or not. If they worked all night and had a lot of fun, they were probably doing dope.”) Smith kept careful note of everyone’s reaction to Superpaint from his vantage point in the color graphics lab, which occupied a long narrow room strategically situated at the nexus of Building 34’s traffic flow—since seven doors opened into it, the passage of personnel rarely ceased.

“Here’s our stuff on the screen, mind-blowing stuff,” he remembered. “Most people would stop and look. Then there were other people who would walk right by and never look. And I’d always wonder, what’s with those guys?”

 

Inside CSL, the person who set the standard of indifference to Superpaint was, unsurprisingly, Butler Lampson. Lampson’s visionary temperament was grounded in a unique pragmatism. He was determined to reach the unseen horizon not by great blind leaps—they posed the unacceptable risk of leading one into a dead end—but by a series of small, measured steps. Big leaps required faith; measured steps required only science and a ruler.

“I remember once having a very illuminating discussion with Butler about my dreams for artificial intelligence,” said Dan Bobrow, the brilliant specialist in computer languages who had been brought to CSL by Jerry Elkind. “He said, ‘Danny, how can you work on something where there’s a goal farther out than two years away?’ Butler’s vision of how you choose projects was to choose those that would tell you in two years if you’d succeeded or failed. He always chose incremental things. I can’t recall him ever having what I think of as a long-term vision. But with his smarts and his good taste he was able to do important next steps in computing and defend them.”

Among Lampson’s objections to color graphics was that it was not by anyone’s definition an incremental thing. “We couldn’t afford color at the time because we couldn’t afford the memory to drive a color frame buffer,” he was still insisting many years later. “I felt you shouldn’t go for it until it’s quite easy, because otherwise it’s going to be a huge distraction.”*

Sure, Shoup acknowledged, color was expensive now, but it would be cheap in five or ten years, just like memory. Why not think of it as just another feature of the Time Machine?

Here entered Lampson’s other important objection to Superpaint: He was constitutionally unable to imagine color contributing anything other than window-dressing to the office of the future. Something so trivial, he argued, might just as well be ignored until it was not merely cheap, but free.

Shoup’s rejoinder was that Superpaint would do much more than enhance the office of the future. “I was looking at a bigger picture: pixel-based imaging in general,” he recalled later. The essential struggle was to get the rest of CSL to see video, color, and animation as not just the technologies behind Saturday morning cartoons and Disney films, but as the foundations of a new type of computer graphics.

Given Lampson’s influence over Taylor and the rest of the Computer Science Lab, this was destined to be a futile mission. Yet the more Shoup sensed himself becoming marginalized, the more he insisted on going his own way. “We attempted to bring Dick into the mainstream, but Dick knew what he wanted to do, and it wasn’t that,” Lampson recalled. As for Taylor, he already considered Shoup an unacceptably reclusive member of a lab he had assembled to serve a shared vision. Instead of joining in the Alto project, Shoup had turned his back. While his own lab colleagues found it hard to work with him, Taylor complained, he constantly gave demos of Superpaint to outsiders—and “non-technical” outsiders to boot, like Smith’s circle of artists and hippies.

As Shoup understood, once you fell out of favor with Bob Taylor there was no coming back. Taylor’s shit list was a cold, forbidding place. He made a few half-hearted attempts rebuild his burned bridges. After the Alto was up and running, he rigged one with a color display. But it was the only color Alto ever seen at PARC and remained forever an object of indifference to most of the engineers in CSL. (Kay’s Learning Research Group, always more highly attuned to the content rather than the process of graphical displays, eventually made excellent use of it.) With every year that passed, Shoup’s performance appraisals sounded more sinister. “Dick,” one read, “is going to have to find a new home.”

One day Taylor walked into the video lab to find Shoup’s equipment festooned with handwritten signs warning: “DO NOT TOUCH WITHOUT MY PERMISSION.” To a manager whose most profound conviction was that his people were all building components of a single common system, this was anathema. He became determined to show Dick Shoup who really owned his precious equipment. One day in late 1974, while Shoup was out of town, he fired the first shot.

The occasion was the broadcast of a television program about the artistic avant-garde entitled Supervisions, which was produced by the Los Angeles public television station KCET. Smith’s and Shoup’s work on Superpaint had started to win wide notice outside PARC, thanks in part to a tape called “Vidbits” which Smith had compiled from clips of his best work for playing to artists’ gatherings all around California. After one such showing, KCET commissioned the two of them to supply some brief color-cycling effects for Supervisions. They had scrupulously insisted that the producers give Xerox screen credit, assuming that the parent company would appreciate the honor.

Instead, Taylor marched into the video lab a day or two after the broadcast and buttonholed Smith. “Xerox wants their logo off every piece of tape,” he said. “Right now.”

He ordered Smith to screen for him every snippet of videotape in the lab—miles of tape. While Taylor sat next to him for an entire afternoon, Smith laboriously ran every reel, including every copy of his own “Vidbits,” punching the ERASE button to excise any frame bearing Xerox’s name or trademark. When Shoup returned home he and Smith managed a nervous chuckle over the sheer absurdity of the incident. But in their hearts they knew it presaged worse trouble to come.

Sure enough, a few weeks later, Smith was dismissed—or more precisely, his purchase order was canceled. The word came from Jerry Elkind, who was nominally Smith’s boss but had never even spoken to him before. “We’ve decided to go with black and white,” he said. “This project is over.”

Smith was stunned. “You’re crazy!” he blurted. “It’s going to be all color from here on out, and you guys can own it all! I can’t believe you’re shutting it down.”

“Well,” Elkind replied evenly, “it’s a corporate decision.”

Smith had no choice but to leave. With a fellow artist and Superpaint fanatic, David DiFrancesco, he drove off toward Utah in quest of permission to continue his work on a frame buffer installed at the university there. He failed to get it, but instead received an invitation to set up a video program at the private New York Institute of Technology. The department later transferred en masse to George Lucas’s Lucasfilm and even later was spun off as Pixar, the studio that produced the hit computer-generated movies Toy Story and A Bug’s Life.

Meanwhile, at PARC Shoup now stood as a solitary pariah. One morning on his way into the lab he was stopped by a sympathetic colleague, who told him: “You know, there’s a meeting going on about you.”

Shoup burst into Taylor’s office, interrupting a discussion about dismantling and redistributing his video equipment to other projects. The group fell sheepishly silent until, clearly unwelcome, “I went down to my lab and waited,” he recalled. A short while later the verdict arrived: His lab space was to be taken away. He was to pack up his taping and recording equipment and turn it over to the PARC audiovisual crew, which would use it to compile a taped archive of administrative meetings.

 

Shoup’s eviction from CSL was answered by a rescue effort by the System’s Science Lab, which secured him a transfer into Kay’s group and permission to reassemble most of his equipment.

But the computer side of PARC never really embraced color as an integral part of its mission. Within a couple of years, when it became clear that Xerox would not support his work on another generation of video graphics, Shoup left PARC. Forming his own company, Aurora Systems, he developed a commercial system that produced the first animated TV weather maps and video logos.

The final irony came in 1983, when the National Academy of Television Arts and Sciences awarded a technical Emmy jointly to Dick Shoup and Xerox Corporation in recognition of Superpaint’s role as a pioneering technology of video animation. Shoup went to the ceremony in New York, where he sat at the honorees’ table with his invited guest Alvy Ray Smith and a nameless functionary dispatched by headquarters to accept the award on the company’s behalf. The television academy had the foresight to prepare two Emmy statuettes. Shoup took his home. After spending a cordial evening with Shoup, the staff man took the other with him back to Stamford, where it vanished into the corporate archives. “I never did find out what they did with it,” Shoup said.

 

*“I feel the same way today about 3-D, which is that for most applications of computing it’s quite marginal,” he added. (This conversation took place in December 1997.)