WHEN KYLE MACHULIS DROVE INTO the parking lot of the San Jose DoubleTree in January 2009, the people lined up for the fursuit parade represented just one-fifth of those attending the Further Confusion conference. A furry, to use a simple and broad description, is a fan of anthropomorphic animals. Most furries create characters, usually cartoonish animals with human characteristics, whom they inhabit online or in real life. The San Jose gathering was among the largest of its kind, largely because furries formed a surprisingly large percentage of Silicon Valley’s tech workers. Getting online was the main way for a furry to connect with her fellows, and when the subculture arose in the late 1980s, that meant knowing how to run servers and build websites. When the dot-com bubble seized Silicon Valley in the mid-1990s, the skills furries had learned to connect with each other became valuable to companies with chairs to fill. The economic frontier of Silicon Valley embraced unconventional lifestyles. Nonconformist ways of living often come with nonconformist ways of thinking, and nothing gets upended—or disrupted, to use the local lingo—by people who think like everyone else. More to the point, a worker’s skill with a keyboard mattered more than however they spent their free time. For all those skills, though, few of the furries in San Jose that day knew what they were posing for as Machulis drove around them in a Nissan pickup whose bed carried what looked vaguely like tornado-tracking equipment. Lidar was still a little-known technology.
A few feet above the truck bed, mounted on a metal trestle, the coffee can–shaped Velodyne Lidar twirled, bouncing pulses from its sixty-four lasers off whatever was nearby. Simultaneously, a GPS tracked the truck’s position, an inertial measurement unit noted its direction, and wheel encoders reported the vehicle’s exact speed. They all piped their results into a bright yellow box, about a foot cubed and sitting next to the Velodyne, which correlated each bit of data to the millisecond at which it was produced. Back in his office, Machulis transformed the returns into a digital, three-dimensional recreation of whatever he had seen. It was detailed enough to pick out the pleats in Minnie Mouse’s skirt.
Machulis had borrowed the truck from his employer, a mapping and robotics company named 510 Systems. Based in a brick-facade, two-story building in downtown Berkeley (and named for the city’s area code), 510 had a lot in common with the other tech startups of the time. It was staffed mostly by young men, had no concept of an HR department, and took a liberal attitude toward how employees should file expenses. Staffers who were into rock climbing scaled the rafters; others brewed beer in the office. They took turns riding through the office on a “haunted” Segway with a tendency to glitch and buck its rider. But 510 had a few unusual traits. It was across the Bay from San Francisco, an hour’s drive from Palo Alto and Mountain View, the Silicon Valley cities where many new companies set up shop. It dealt with complex hardware, when many were getting rich by developing mobile games like FarmVille and Candy Crush. It divulged little of its workings to outsiders. It included so little information on its website that one job candidate thought 510 must do some sort of secretive defense work. And its CEO was a woman in her mid-fifties named Suzanna Musick, whose background was in marketing. But she didn’t set the company’s direction or priorities. The real boss was her stepson, who had moved in with her and his father as a teenager, after spending his childhood in Belgium.
Anthony Levandowski founded 510 Systems with fellow Berkeley grads Pierre-Yves Droz and Andrew Schultz on May 10, 2007. Levandowski at the time was working for Sebastian Thrun on Google Street View, helping on Stanford’s Urban Challenge team, and selling Velodyne Lidars for Dave Hall. After the Urban Challenge, he took a job studying unmanned technology with Ensco, the defense and aerospace contractor whose bathtub-like robot had flipped over sixty-six seconds into the 2004 Grand Challenge. He soon tired of the work, uninterested in writing reports that, if successful, would land him a contract to write more reports. Though he kept his name off much of the 510 paperwork, this new company was his creation. And it was making technology that promised to change how its customers saw the world.
Millions of Lidar returns, camera images, and GPS coordinates aren’t much good for mapping unless you can link them together, the way the human brain automatically links what its body sees, smells, hears, tastes, and touches into a cohesive understanding of its surroundings. That’s what 510 Systems did. It developed the IP-S2, or Integrated Positioning System. This was the yellow box amid the sensors on the truck Machulis took to the DoubleTree in January 2009. It collected and correlated the data from those sensors. 510 wrote the software that took the results and turned them into a 3D visualization a person could inspect and analyze. The result was a powerful new tool for creating a high-fidelity record of a physical space, to be used by anyone who needed to know where things were and how they moved over time.
With that kind of intel, utility companies could see where tree branches threatened their power lines. Transportation departments could keep track of road signals; construction crews could measure their progress at work sites. Conventional surveying techniques involved workers setting up tripods in carefully selected locations and collecting data point by point. With the IP-S2, anyone could instead mount the surveying rig on a vehicle, drive through the area of interest, and click through the 3D map that 510’s software spat out.
To put the tool in customers’ hands, Levandowski paired with Topcon, a Japanese company founded in 1932 to make surveying instruments, binoculars, and cameras for the Imperial Army. After World War II, Topcon went civilian, eventually expanding into California and making a variety of high-tech tools for the medical, construction, and agricultural industries. Levandowski had first reached out to the company in his search for sponsors before the 2004 DARPA Grand Challenge. Topcon exec Eduardo Falcon was charmed and gave Levandowski his first cash donation, of $20,000, along with a GPS. No matter how well the Blue Team did, Falcon figured, they would stand out for having the only motorcycle. When Levandowski came back a few years later with this venture, Falcon was happy to reconnect.
For the IP-S2, 510 Systems worked as a sort of external R&D lab, figuring out how to make the thing work and customizing it for different uses. Topcon would manufacture and sell the result, promising customers what it dubbed the “continuous representation of reality”—a detailed, accurate, and near real-time portrayal of whatever it was they wanted to see. Customers for the “Topcon box” included Microsoft, which was working on its own location imagery technology. Nokia-owned mapping company Navteq used the box to develop a turn-by-turn navigation offering. One of the biggest buyers was Google, which used the IP-S2 for Street View. Which was odd, because Levandowski hadn’t left Google when he started 510. He was still working on the hardware side of Street View, and now he was sending his employer’s money across the Bay, into his own pockets. The Google bosses, including Sebastian Thrun, knew about this unorthodox setup, but at one of the world’s most profitable companies, who cared about the cash? Levandowski, as hard-charging as ever, was getting the job done.
The workers at 510 knew Levandowski worked for Google, but didn’t have much in the way of details. Campbell Kennedy, a 510 product manager, understood a bit more when the Street View team was having trouble with the IP-S2 and asked if someone from 510 could help them out in person. He went to Mountain View with Levandowski and Suzanna Musick. When the three walked into the room for the meeting, Levandowski sat down not with them, but with the Googlers—the Googlers who then grilled Kennedy about 510’s technology.
Levandowski’s Google commitment—including regular trips to Hyderabad, India, where two thousand workers turned the Street View cars’ data into useable maps—kept him away from 510 for long stretches. When he was around, usually in the evening, he had the same effect on his employees he’d had on his DARPA Challenge teammates. He was charismatic, full of ambitious ideas, and unimpressed by practical hurdles. One day, 510 software engineer Ben Discoe was sitting in the office kitchen, musing over how great it could be to have a 3D visualization of his family’s Hawaii tea farm. Levandowski wandered in, overheard, and told him to just go do it. A few weeks later, Discoe was flying west with tens of thousands of dollars’ worth of 510 equipment. Another engineer, Willy Pell, found conversations with Levandowski almost addictive. The 510 founder understood their tech at the detailed level, but could also weave disparate ideas into big-picture imaginings. They’d have long brainstorming conversations, covering whiteboards with ideas. This was where Levandowski was at his most compelling: in small groups where he could connect with people one-on-one, waving his long arms and urging others to debate with him, may the best idea win.
It made for an exciting atmosphere but an inefficient business. The company engineers reported sporadic direction from the top, and were moved from project to project at a moment’s notice. One post on the business-focused reviews site Glassdoor complained, “Not much information from management about the direction of the company.” Another read, “Uncertainty and doubt as to what is going on. Management does not tell the truth to employees. Overall direction of the company is unknown.” Pell compared 510’s structure to a subway map’s tangled, intersecting lines. They had hardware and software efforts, and a stable of customers with particular needs. One team built a version of the sensing rig that fit into the belly of a Cessna airplane, so Google Maps could include 3D images of cities taken from the air. On the building’s second floor, 510 cofounder Pierre-Yves Droz started developing a new Lidar, to end the company’s reliance on Dave Hall’s $70,000 sensors. The company helped produce the music video for the Radiohead song “House of Cards,” which was shot in Lidar. Its engineers helped Topcon develop controls systems for automated farm equipment. Yet another node on the subway map shared a building and some employees with 510, though it in fact belonged to another company, which Levandowski had incorporated on June 12, 2008. On the second floor of 510’s Berkeley office, Anthony’s Robots was building a car that could drive itself.
For Levandowski, the conclusion of the DARPA Urban Challenge in November 2007 was just a starting point. From his first days building his robotic motorcycle, he understood that autonomy could fundamentally change how everyday people moved through their lives. So when a producer from the Discovery Channel asked him if he could use his autonomous motorcycle to cart a pizza through San Francisco, Levandowski wanted in. He knew the producer from his 2007 appearance on the Animal Planet show Chasing Nature, in which student engineers mimicked the skills of various animals. In an episode focused on the jumping ability of dolphins, Levandowski traded his “Blue Team” T-shirt for a wetsuit, flippers, and a back-mounted air cannon. Towed behind a boat, he fired the cannon and flew twenty feet through the air before splashing down, face-first. No other contestant soared farther.
Now the producer was working on a Discovery Channel show called Prototype This! In each episode, four “inventors” took on an outlandish challenge by building outlandish machines. In their one, thirteen-episode season, they made a flying mechanical lifeguard, a wearable airbag for workers building high-rises, a water slide simulator, and a pair of huge boxing robots. They filmed the show on Treasure Island, a four-hundred-acre man-made landmass in the San Francisco Bay that the Army Corps of Engineers had built to host the 1939–40 Golden Gate International Exposition, with the idea to convert it later into an airport for the Pan Am Clipper seaplane. The navy took it over during World War II and stayed until 1997. In 2008, Treasure Island was a lonely place, home to some low-income housing and a legally acceptable amount of radiation deposited by ships exposed to nuclear testing during the Cold War. But it was the distance from San Francisco—you had to drive half the length of the oft-congested Bay Bridge to reach the island—that made it hard to get food delivered there. This genuine source of frustration for the Prototype This! crew was the premise for the show’s eighth episode, “Automated Pizza Delivery.” Over the forty-five-minute episode, the gang tried tossing pizzas out of a low-flying blimp and customized a rolling robot to carry the cheesy pies. Then they turned to “software engineer, autonomous vehicle expert, and all-around whiz kid Anthony Levandowski.”
The show’s narrator didn’t mention that Levandowski played a role in Street View, one of Google’s most audacious and beloved programs. Levandowski had brought the Prototype This! pitch to his bosses for approval. Google didn’t want its name anywhere near testing unproven tech on public roads for some goofy show. He could go ahead, but was to keep his relationship with the tech giant out of it.
The producer’s original idea was to use Levandowski’s autonomous motorcycle, but Ghostrider wasn’t available. Along with Stanford’s DARPA Grand Challenge–winning Stanley and Dave Hall’s original Lidar, it was now part of the Smithsonian’s permanent collection, officially sanctioned as a piece of technological history. Levandowski used two Toyota Prius hybrids instead. The first was equipped with a suite of lasers and the 510 Systems–designed IP-S2. With Levandowski doing the driving, it mapped the delivery route from downtown San Francisco, onto and across the Bay Bridge, then down the ramp onto Treasure Island. The second car was the star of the show. In a Berkeley driveway not far from the one where he and his Blue Team had worked on Ghostrider, Levandowski showed off Pribot, with a rooftop Velodyne Lidar and a trunkful of computers.
The Prototype This! narrator didn’t reveal that just a few weeks before the shoot, Pribot hadn’t existed. It was the product of a feverish push by Levandowski, his 510 cofounder Droz, and 510 product manager Campbell Kennedy. In a throwback to the DARPA Challenges, they went days without sleep, racing to make their robot work before the fateful day when it would be put to the test. They started by tapping into the Prius’s CAN bus, effectively the car’s nervous system, which would let them control it with electronic commands. That worked fine for the acceleration and steering, but reverse engineering the braking proved trickier. The Toyota software that ran the electronic stability control and antilock braking made the effort so complicated and time-consuming that they opted instead to install a hydraulic motor on the floor of the car to physically push the pedal. Without a proper proving ground, they tested their work on the roads of Berkeley at night, always with someone in the driver’s seat, ready to hit the kill switch.
The hastily built robot, with a 510 Systems logo covering its hood and “Topcon” written across its bumper, could not read traffic lights or detect stop signs. It couldn’t have handled the traffic that faced the bots in the Urban Challenge. Since it only had a rudimentary collision detection system, it might not have been able to complete the simpler Grand Challenge, either. It only stood a chance of making the journey through San Francisco because the Discovery Channel producers arranged to have the police shut down the streets that made up its route, including half the Bay Bridge.
Challenge day came on Sunday, September 7, 2008. A little before eight in the morning, Levandowski set up Pribot on the Embarcadero, the boulevard that follows San Francisco Bay’s curving shoreline. The car hesitated as it pulled away from the curb, then put itself in the middle of its lane, its steering wheel twitching and its Velodyne Lidar spinning. It followed GPS waypoints, much like the DARPA bots that had competed in the Mojave. This robot, though, traveled in the middle of a police motorcade and right behind a flatbed truck, from which Levandowski looked back, ready to hit the remote kill switch. It wasn’t the smoothest performance. Pribot strayed to either side of its lane at points and took wide turns. But it slowly made its way through the city and up the winding ramp that led to the lower, eastbound deck of the Bay Bridge.
Holding the kill switch in his right hand and high-fiving the show’s hosts with his left, Levandowski watched his creation cross the span. But when the car reached the exit for Treasure Island, the cheers turned to groans. As Pribot curled down the hairpin ramp, it edged left, scraping against the concrete railing and pinning itself stuck. Levandowski shut it down, folded his lanky body through the driver’s side window, and drove the car the rest of the way off the ramp. Once safely on Treasure Island and back in autonomous mode, Pribot made the final few turns into the Prototype This! office parking lot. Each hoisting a slice of freshly delivered pizza, Levandowski and the show’s hosts declared a toast “to the first ever autonomous pizza delivery vehicle.” More than that, it was, almost certainly, the first time an autonomous vehicle had explored public roads with nobody behind the wheel.
Soon, though, Levandowski would be part of a much bigger, more serious effort to deliver on the promise of a technology the DARPA Challenges had shown was possible—one that wouldn’t end when the credits rolled.
Larry Page had been interested in autonomous vehicles long before he attended the DARPA Challenges. As a twenty-two-year-old student in Stanford’s computer science doctoral program, he’d considered the technology as a research idea. Instead, he followed his advisor’s advice and focused on search, finding a new way to understand the links between the disparate regions of the World Wide Web. A decade after founding Google with his classmate Sergey Brin, he was starting to push his company’s influence beyond the limitations of screens and keyboards. Sebastian Thrun (aided by Anthony Levandowski) had proven an effective conquistador, accelerating the Street View project and then spearheading an effort called Ground Truth, in which Google built its own global database of maps from scratch. The massive project, almost unfathomable in its ambition and relying on the work of thousands of people, freed Google from the need to license the data of existing cartography companies and formed the basis of all its ensuing mapping efforts. It also gave Thrun what he called “an appetite for scale” that drew him further and further from his past life as an academic.
Now Page wanted to put computer-controlled cars on the streets his company had mapped. The influence of the Prototype This! episode on his thinking is hard to discern. Some insiders believe the creation of Pribot was a bid to push for a robotics project at Google, a public stunt by Levandowski and his boss, Thrun, to show Page they intended to tackle autonomous driving one way or another.
In Thrun’s telling, it was the opposite: that Page was intent on pursuing the tech, and Thrun tried to wave him off. The DARPA Challenges were one thing, with their limited conditions. No traffic lights, no pedestrians, no unexpected construction crews, no crushable tortoises. Trying to build a robot that could navigate the real world, in all its complexity and chaos, was folly, Thrun said. Page persisted, asking for a technical reason why it was impossible. Frustrated, Thrun snapped. “It can’t be done, goddammit,” he said. Then he went home, tried to think of that specific, technical reason, and came up empty. When he admitted as much, Page talked up the economic view that Levandowski had grasped heading into the first Grand Challenge: Transportation formed an enormous chunk of the economy, and applying autonomy to even a sliver of it could make for a massive business. They could make a company as big as Google itself, Page said. Even if it had a one-in-ten chance of working, it was worth the investment of money and time. Whatever the exact motivation, Thrun agreed to give it a go, and he knew exactly who he wanted to join him. He had met them in the Mojave Desert.
In October of 2008, six men met Thrun at his Lake Tahoe home. All were veterans of the two teams that had dominated the three DARPA Challenges. From Thrun’s Stanford effort, there was Mike Montemerlo, Hendrik Dahlkamp, Dirk Haehnel, and naturally, Levandowski. Chris Urmson and Bryan Salesky represented Carnegie Mellon. As rivals, they had pushed one another to create some of the world’s most sophisticated autonomous vehicles. Now Thrun wanted them to come together as one team and finish what they had started outside the Slash X Cafe.
Google, though, seemed an odd place to do it. It was a search engine company. It did email and maps and owned YouTube, but it didn’t make any hardware. (That changed in June 2012, when Google introduced a digital media player called the Nexus Q. Today it makes phones, tablets, laptops, and more.) Street View and Ground Truth charted the physical world, but they didn’t interact with it the way an autonomous car would. This would be a totally new direction for the company. And how could the DARPA veterans be sure Page wouldn’t lose faith in this project after a year and kill it? They’d be left without jobs, having abandoned promising, dependable careers for a risky Silicon Valley venture.
Thrun had counterarguments. Google might be about software, but it was no ordinary company. When it had gone public in 2004, Page wrote a letter warning potential investors that he and Brin had no intention of bowing to short-term financial objectives. “We may do things that we believe have a positive impact on the world, even if the near term financial returns are not obvious.… We will not hesitate to place major bets on promising new opportunities.” Google was awash in cash—it made more than $4 billion in profits in 2008 alone—and could afford to fund the effort, even if no one knew what it would cost or how long it might take. And in case Page did give up on the idea, Thrun designed a compensation package with lavish (especially compared to the academic world) salaries and bonuses that would help them get through any stretch of unemployment. If they did their work well, the team’s early members would make upwards of $1 million in a year.
Most of all, Google was offering them the chance to be pioneers. Even the roboticists among them hadn’t imagined working on cars before Tony Tether started talking about a race in the Mojave. After the Urban Challenge, Chris Urmson and Red Whittaker had proposed to General Motors that they continue their autonomy partnership, building on what they had done in the mock city. The Detroit automaker declined. Even a year before the financial crisis that would knock it into bankruptcy, GM’s business was imploding. In 2007, it lost $38.7 billion, an auto industry record. Perhaps the only company pursuing fully autonomous vehicles was Caterpillar, which had hired Urmson and Salesky to develop robot trucks for working in mines.
Thrun offered these engineers the opportunity to do something they and everybody else considered just a bit nutty. To take their DARPA Challenge work out of the desert and put it in the middle of the world where they lived. To give them a few stories to tell their grandkids.
Thrun’s sales pitch wasn’t enough to convince Salesky, who wanted to see the Caterpillar project through and wasn’t ready to abandon Pittsburgh for Silicon Valley. Thrun did win over most of his Stanford teammates, who were already in California, most of them working for Google in some capacity, and Chris Urmson, who was game for a change of scenery, a new mission, and a promotion. Thrun would be the head of the projects, but when it came to the day-to-day work of teaching a car to drive, Urmson would be in charge, if unofficially. He resigned his role on the Caterpillar team, gave up the professorship he had just started at Carnegie Mellon, and headed west.
Google’s vision was grand in its ambiguity. Page wasn’t proposing a typical government or defense industry contract. He wasn’t looking for a specific solution to a specific need, with carefully defined funding and spending limits. He just wanted an autonomous car, and he was willing to write something that looked a lot like a blank check to get it. He did, however, believe in aggressive goals. In negotiations with Thrun, he and Brin set two targets for the team, one to focus on scale, the other on skill. The first was to accumulate one hundred thousand miles of autonomous driving on public roads. Those, they could log wherever they liked. Then the cofounders pulled up Google Maps and selected ten routes, each roughly one hundred miles and all within California. Thrun and Urmson’s team would have to make a car that could handle every inch of each route, without human intervention, about a thousand miles in all.
“They took great pleasure in picking really hard roads,” Thrun said. Roads like San Francisco’s famously double-jointed Lombard Street. Roads through downtown Los Angeles and the Sierra Nevada mountains. Roads tracing the winding California coast. Roads that added up to a thousand miles of cruelty, making the competitions set forth by Tony Tether and Sal Fish look laughable. At first, Thrun griped that it was impossible. If they’d just allow him one or two human interventions per route, it might be doable, he said. Page and Brin, echoing the DARPA director’s thinking from 2003, refused. It’s only worth doing if you can take the human out of the picture altogether. The team called this new challenge the Larry 1K. They had two years to reach the finish line.
On a Monday morning in early February of 2009, Chris Urmson found himself standing next to Dmitri Dolgov, each waiting in line to get his new employee badge. The men knew each other mostly by reputation. The Russian-born Dolgov had earned a PhD in computer science from the University of Michigan and, as a postdoc under Sebastian Thrun, had become a key member of Stanford’s Urban Challenge team. Now the engineers were on the same side, along with a murderer’s row of old rivals and new teammates. Thrun brought along his old Stanford allies and new contacts he’d made at Google; Urmson came in with his fellow CMU alum Nathaniel Fairfield. All together, these eleven men accounted for some of the best young engineering talent in the country. (Indeed, the world: The majority were foreign-born.) Thrun, Urmson, Levandowski, Mike Montemerlo, and Dirk Haehnel—all of them DARPA Challenge veterans—were the official cofounders of the team, with particularly rich incentives. They believed that they had the skills to expand what they had done in the Urban Challenge to the real world. Very much a secret effort outside of Google (and even within it), they adopted a code name: Project Chauffeur.
Their effort to create what they termed the “self-driving car”—replacing the more common “autonomous vehicle” of the DARPA Challenge era—started by licensing the rights to the code Stanford’s team had used in the Urban Challenge. It wouldn’t be anywhere near good enough for the task at hand, but it would give the team what Urmson called a “jogging start.” Instead of starting from scratch, they had something to improve. And the engineers tackling different aspects of the robot—perception, planning, and so on—could all dive in at once. It worked because they would follow the basic formula for a robot that the DARPA Challenges had put forward, with a drive-by-wire system implementing commands from a computer that saw the world with radar, cameras, and a roof-mounted, spinning Lidar system developed by Dave Hall.
The engineers set up in Building 1950, a curving glass structure on the eastern edge of Google’s campus that also held Larry Page’s office. Mapping whiz Montemerlo—who kept the roughly life-sized eagle trophy for Stanford’s second-place finish in the Urban Challenge on his desk—took on the code that would teach the car to locate itself in the world. Dolgov developed the motion control system. Street View veteran Jiajun Zhu did perception; Fairfield worked on traffic light detection. Urmson ran the show from day to day, but as with many small teams tackling huge problems, Chauffeur had a tendency toward egalitarianism.
Life at Google had little in common with the research universities from which most of the Chauffeur team came. The money was better. For those coming from Pittsburgh, so was the weather. Lunch was free, as were breakfast, dinner, and snacks. Employees had access to gyms, workout classes, and massages. More importantly, Google offered the technological infrastructure to deal with huge amounts of data and code. But the thing that most struck Urmson was how empowered his team was. He’d spent most of his professional life in universities and was used to research grants with persnickety rules governing details like which pots of money could go to which equipment. Here, if he needed a new GPU, Lidar scanner, or car, he just ordered one. Meanwhile, Urmson settled into life in the Bay Area, where palm trees jockeyed for space with redwoods. He made the move a few months before his wife and two young sons, and in the interim accepted a generous offer from one of his new teammates: Anthony Levandowski knew Urmson casually, and offered him a spare room in his Palo Alto house.
One of the few team members without a PhD, Levandowski worked on hardware, as he had on Google Street View. He went to a Toyota dealership in San Jose, bought half a dozen Priuses, and retrofitted them to accept a digital overlord. Unlike with the car that crossed the Bay Bridge for Prototype This!, he figured out how to properly tap into the brake electronically, along with the steering and accelerator. Then he attached the sensors that would be their guides. A radar behind the front bumper watched the road ahead. One camera faced forward, another scanned 360 degrees. And atop a roof rack sat one of Dave Hall’s spinning Velodyne Lidars. The team gave the cars names drawn from pop culture like KITT and KARR from David Hasselhoff’s Knight Rider, and Car 54 (a reference to a 1960s sitcom about New York City cops).
As the cars came together, Levandowski bounced between Google and 510 Systems. His own company—most everyone on the team knew he owned it, and that the higher-ups knew too—served as a contractor, doing some of the hardware work. Levandowski moved the cars between Mountain View and Berkeley, even storing various parts in his own garage. And as ever, he delivered at remarkable speed. In the DARPA Challenges, teams had typically taken months to get a vehicle up and running. Levandowski put half a dozen cars together in a couple of weeks. “If you need to blow up a dam in Germany, Anthony’s your guy,” said Isaac Taylor, Chauffeur’s operations manager, referring to Barnes Wallis, the English engineer who invented the bouncing bomb RAF pilots used to destroy dams behind enemy lines. “The general who’s going to win you World War II.”
The differences between Levandowski and Urmson were clear from the beginning. The roommates got along well, but they came at the problem—at the world in general—from opposite mind-sets. Where Levandowski moved as a blur, Urmson was methodical. Near the start of the project, he told Thrun he needed to rewrite the car’s core communication software. Thrun, who leaned toward the fast and scrappy side, thought the system they had was fine. Even if it crashed occasionally, stopping to rework it could slow progress. Urmson insisted, and took several weeks to do the work. The result was a stronger, more reliable system. “He was damn right,” Thrun later admitted.
The team started off testing in the parking lot of the Shoreline Amphitheatre near Google HQ, where Stanford’s Grand and Urban Challenge teams had worked, and at NASA’s Moffett Field a few miles down the road (yet another old military base pressed into service). But conquering the Larry 1K meant taking the inherent risk of driving on public roads, those streets and highways filled with everyday humans, and without the benefit of the police escort that had made the Prototype This! run through San Francisco possible. After starting 2009 by testing in private—with a security guard posted at the entrance to the parking lot—establishing the car’s ability to execute a computer’s commands and understand Lidar data well enough to guess what was a car and what was a human, they took their first few cautious steps into the real world in the spring.
As a warmup to the nasty, hundred-mile routes that made up the Larry 1K, Chris Urmson started the car on Central Expressway. Running alongside train tracks, the Silicon Valley thoroughfare was four lanes wide, with a grassy median separating northbound from southbound traffic, a forty-five-mile-per-hour speed limit, scarce pedestrians and cyclists, and mile-long stretches between the traffic lights the car did not yet know how to interpret. It was a relatively simple learning environment.
Before the road could welcome the robo-cars, however, it had to be mapped. If the Urban Challenge had imparted two practical lessons, they were that Lidar was vital, and good maps made moving through a complex, dynamic world a whole lot easier. Even for the teammates who’d spent years on Google’s mapping projects, this was cartography on a new scale. Driving their Lidar-equipped cars, they catalogued the precise location of every lane line and curb, the signs and the traffic lights, all of it accurate down to ten to fifteen centimeters. That precision offered two key benefits. First, it took some strain off the car’s ability to perceive its surroundings in real time. With an a priori knowledge of where it would encounter stop signs, speed limits, and the like, it wouldn’t steam through an intersection if the sensors missed the red octagon, or risk a ticket if they didn’t spot the school zone speed limit. Second, it made it much easier to pinpoint the car’s location. GPS was accurate to within a few yards at best, an unacceptable margin of error for driving in a populated world. If the Lidar said the car was ten feet from that speed limit sign, and twenty feet from that light pole, and the map told it exactly where those things were in space, it could do a quick spot of math and deduce its own location. This concept of mapping and localization is not unlike the way people move through their house in the dark. They reach for landmarks like the corner of the desk or the bedpost to check where they are, and arrive at the light switch without stubbing a toe.
Maps in hand, the Googlers sallied forth, and relearned another lesson from the DARPA Challenges: how much they could accomplish when they were allowed to focus on nothing but making a car drive itself. They had no priorities but the Larry 1K. The tiny team engendered no bureaucracy, no pointless meetings. They reported directly to Larry Page. “How beautiful it was not to do email every day,” Thrun said. And as they went from the old Stanford code made for the controlled environment of the Urban Challenge to something that could tame the wilderness of real roads, they saw tangible progress from one day to the next. Chris Urmson and Dmitri Dolgov could take the car out in the morning, see it fail to make a given turn, make some code changes after lunch, and that evening watch the car nail that same move. It wouldn’t do it perfectly. It wouldn’t do it every time. But with each code change, it got better, more reliable, more human.
After those first runs on Central Expressway, the team decided it was time they started making real progress toward their goals. Their tech still wasn’t good enough for the wicked roads of the Larry 1K, but it could take on the environment that would move them toward the other goal Thrun had negotiated for them, logging one hundred thousand autonomous miles. Driving on the highway required higher speeds, and a mistake could be deadly. But compared to surface streets, it was easy to navigate, with all the cars going in one direction and no intersections, cyclists, or pedestrians to worry about. They started with a sixty-five-mile loop of Bay Area highways that took about an hour.
Instead of putting engineers behind the wheel on a regular basis—their time was better spent in front of a computer—Chauffeur fleet operations manager Isaac Taylor hired third party contractors. For $25 an hour (plus overtime and expenses), they would work as “safety operators,” a new term for the role of riding in the driver’s seat while the car drove itself, ready to take control from the computer if a crash seemed possible. Without a model to emulate—no one had ever tested a fleet of robots this way—Taylor created a new kind of testing regimen. Operators would go two to a car. One sat ready to take control if necessary, his right hand always near the big red button that would sever the computer’s control of the car. (Hitting either pedal or turning the wheel had the same effect.) The other rode shotgun with a laptop, looking at the graphical user interface Dolgov created to turn the car’s software and sensor data into something a human could understand. The car automatically created its own record of what it was doing, which the human annotated, noting where it drifted from its lane, when it braked for no apparent reason, how it handled turns, and any time the driver disengaged the autonomous system.
New safety operators went through a weeks-long training that started with a week in the back seat of a car worked by two seasoned hands up front. They were to observe only. No talking, no risking distracting the operators from their jobs. The only thing they were allowed to say was “disengage”—if anyone said the word at any point, the driver took control. After training, the operator could graduate to the right front seat of one of the cars building the high-res maps that so helped the self-driving system. Their first rides in one of the autonomous cars would be with one of Taylor’s most experienced drivers. Eventually, they’d be tested by Taylor, Urmson, or another high-ranking Googler, who’d watch them work the laptop and make sure they knew how to produce data the engineers could learn from.
Before getting behind the wheel, they’d go through a rigorous course that covered how the system worked and how to handle a vehicle that went out of control. On a regular basis, Taylor, a former amateur rally racing mechanic, would bring in pro driving instructors and run clinics for new drivers. Taking over the trusty Shoreline Amphitheatre parking lot, they’d go through the kind of training police get, learning to handle emergency lane changes, recover from skids, pull over safely with a blown tire, and so on.
Safety operators worked eight-hour shifts that typically involved six to seven hours of driving time. Before leaving base, they went through a carefully detailed checklist, making sure the sensors and software were running properly, the computers in the trunk were good, the taillights and headlights worked, the tags were accurate and up to date, the drivers were well rested, and more. They were encouraged to take breaks when they needed a moment’s rest, and were free to expense any snacks they picked up along the way. By October of 2009, Taylor had built a team of twenty operators, along with a few folks doing basic maintenance on the cars.
As the fleet started logging serious miles, the engineers fell into their own rhythm. When the safety operators came back from a run and submitted their data log, a software engineer would pick out an issue that had come up multiple times—an awkward merge, say, or ping-ponging left and right within a lane. The engineer would go over the data, looking for what might explain the problem, then start working on the algorithm in question, writing new code when tweaking the existing stuff didn’t do the trick. Once they thought they had a solution, the engineers would run it through a computer simulation, and then send the software to a car that ran on a closed course. Then it would go to a car plying public roads, with one of those most experienced safety operators. If the problem persisted, the engineers would try another approach. If it went away, they dropped their bucket back into the well of problems, pulled up the next issue, and started the process over again.
They broke up the coding work with regular games of foosball. Several times a day, the Googlers would assemble, two or three on each side of the table. They’d pull, push, and spin the handles, moving the plastic footballers through time and space. But even in this smallest, simplest facsimile of the real world, no one could make the ball do exactly he wanted—not every time, anyway.
As the team got deeper into the spring of 2009, Urmson decided it was about time they started going after their grand challenge, the Larry 1K. Of the ten one-hundred-mile routes, Urmson settled on starting with the one that went from Carmel-by-the-Sea on the Monterey Peninsula to the coastal town of Cambria, via California’s famed Highway 1. The winding, narrow coastal road might spook a human driver, but as Stanley, Sandstorm, and H1ghlander had shown on the 2005 Grand Challenge’s Beer Bottle Pass, robots don’t mind such things. Rather, this route looked to be easiest of the ten, because it included just a handful of intersections and traffic lights. The bulk of the drive would be spent sticking to a lane and not veering into oncoming traffic.
Before that, though, the caution-minded Urmson set up another task: making the robot run a four-mile stretch of a Highway 101 frontage road, including a few traffic lights, ten times without the driver needing to take over. Once they’d done that, he was willing to send his engineers—no contracted drivers for the Larry 1K routes—down to Carmel to start trying the official route. After a mapping car had run the route and Dmitri Dolgov had driven it to preprogram the software for the best path to take, Urmson, Dolgov, and Levandowski climbed in the Prius. Dolgov had programmed the car to slow down very late when going into curves, a bit like a racecar driver. Looking at driving from the standpoint of physics, it made sense. But the engineers quickly realized it wasn’t comfortable for them, especially on a winding coastal road. And if their car was going to someday carry passengers, it would have to drive more like an everyday human.
But they let it carry on, and the car stayed within its lane. All the while, Urmson’s hand hovered over the red disengage button. It finally came down when they encountered construction that cut the road to a single lane. The car wasn’t anywhere near capable of understanding the worker directing traffic, and they weren’t going to chance anything. The trio went back to Carmel and gave it another shot, after Dolgov had tweaked how the car approached turns. By the time they reached the construction site, the road was back to two lanes, and there was nothing to stop them from reaching Cambria. They had covered 101 consecutive miles as passengers in a self-driving car. Exhausted, they celebrated with a beer and dinner, reflecting on what they had done, and what that might mean. “We’re going to make billions,” Levandowski said. After spending the night down south, they headed back to the office and placed an empty bottle of cheap Korbel champagne on a shelf. On the label, they had written the route and signed their names.
Next, they tackled a one-hundred-mile stretch of the El Camino Real, the road built by the Spanish to connect the missions they established along California’s coast. Modernity had transformed the dirt footpath into a proper paved road, complete with busy intersections and, on the stretch that Page and Brin had selected, 237 traffic lights. Those proved easier to handle than the bit of the route that sent them through downtown Palo Alto, where the car’s struggles around pedestrians and diagonally parked cars forced Urmson to hit the disengage button again and again. But each failure gave them something specific to fix, turning a huge, abstract challenge of driving safely into a series of tangible problems to solve. Eventually, another empty bottle of champagne joined the first one on the shelf.
The team really hit its stride in 2010, knocking off Larry 1K routes that took them around Lake Tahoe, south to Los Angeles, up the I-5, and along Pebble Beach’s famed 17-Mile Drive. Chauffeur’s capabilities expanded to handle all sorts of driving scenarios, but still relied on a combination of persistence and luck to guide the car from the start to the end of any given route. The engineers would try over and over, usually making their attempts late at night when traffic was lighter, making endless adjustments to the software. “We attempted them many times until we got lucky,” said Don Burnette, one of the engineers who joined the team as it slowly grew over that first year.
The system at this stage was capable, but its makers were specifically hand-tuning it for whatever route it was attempting at the time: how aggressive to be on a given merge, where exactly to be in the lane at every moment, what degree of color counted as red given the angle of a traffic light and the sun at a certain time of day. Burnette was in the car with Urmson, Dolgov, and Levandowski one night, late in the summer of 2010, when the resulting brittleness struck near the very end of a Larry 1K route. They were charged with traversing the five bridges that cross the San Francisco Bay, and had reached the final one—the Golden Gate—when they realized the toll booth for the lane they had selected was closed. The car’s programming didn’t offer the flexibility to jog to the left or right. Down went the red disengage button.
The next day, they tried again, and this time hit trouble in Tiburon, just before reaching the Golden Gate. They were on a narrow road, with cars parked on both sides, when another driver approached from the opposite direction, then pulled over to let them pass. The robot, though, was programmed to require more clearance than it had. Levandowski got out of the car and asked the driver to move farther over, without being able to say it was a robot car and that if they could just get past, they’d be closer to making Larry Page a happy man. The man was confused and annoyed, and Urmson wanted to defuse a potential problem. Down went the red disengage button. The trio returned to Mountain View, refueled the Prius, and tried yet again—this time making it all the way through. Returning to the office in the early hours of the morning, they put another bottle of Korbel on the shelf and celebrated with a game of foosball.
In the spring of 2010, the team had suffered its first serious crash. Two safety operators were doing a run on surface streets, testing out a new traffic light detection system. On Central Expressway, when a light turned yellow, the car hit the brakes, and was promptly rear-ended by a pickup truck. The robot, a maroon Prius named KITT, was totaled. The two safety operators—and their friend who was definitely not supposed to be sitting in the back seat—were fine, as was the truck’s driver. Nobody called the cops, and the press didn’t get wind that Google was testing self-driving cars on public roads, and crashing.
That crash came a few months after an incident that stemmed from Levandowski’s endless desire to move ahead. As fleet operations manager, Isaac Taylor determined how the cars tested on the highway. While he was on paternity leave, Levandowski—the hardware lead—decided to expand the test envelope to new situations, including driving consistently in the right lane of the highway. A team member alerted Taylor, who returned to the office furious. After arguing, he and Levandowski took a Prius and a nearby truck onto the I-280 to see if Taylor was too cautious or Levandowski too aggressive. Perhaps because the robot’s map didn’t include subsequent on-ramps (part of the reason it was meant to stay to the left), the Prius didn’t make room for a Camry merging aggressively onto the freeway. Levandowski moved to avoid a collision, and the Camry spun out. This event, too, went unreported (until 2018), although Urmson included it in a 2011 presentation to the Institute of Electrical and Electronics Engineers—minus the argument—saying the Camry’s driver ended up with “a little bit of excitement in his day.”
On September 27, Urmson, Dolgov, and their fellow engineer Andrew Chatham got into a Prius to try the tenth and final leg of the Larry 1K. This was a particularly nasty and diverse one, taking them through Palo Alto, up Skyline Boulevard into the Santa Cruz Mountains to the west of Silicon Valley, then down to the Pacific Coast on Highway 1. From there, the route went north into San Francisco, then east through the city. In the Russian Hill neighborhood, it went north and then south four times within a mere 450 feet. Page and Brin had taken great pleasure in picking the hardest roads in California, and they weren’t going to miss San Francisco’s famously crooked Lombard Street. Once again, the car showed that humans and robots have very different definitions of difficulty. A windy street that could stress a human didn’t perturb the Googlers’ software. The true test came when the Prius turned onto Market Street. This major thoroughfare cuts diagonally through the street grid, making for tricky intersections. It carries not just cars and trucks and taxis, but bikes, buses, and a streetcar line. Its wide sidewalks teem with pedestrians. But that Monday, the car handled it all. With the rest of the Chauffeur team tracking the Prius’s progress on monitors in the office, Urmson, Dolgov, and Chatham let the car take them into the hills at San Francisco’s center. Then, high above the city they had just navigated, they reached the end of Market Street, and the end of the Larry 1K.
The moment called for more than a single bottle of champagne. The team, safety operators included, gathered for a grand party at Sebastian Thrun’s house in the hills of Palo Alto. They threw one another into the pool and played “human foosball” on an inflatable field. The celebration was well deserved. This team of all-star engineers had taken on a challenge that made Tony Tether’s DARPA races look simplistic, and they had conquered it in well under the allotted two years. Almost impossibly quickly.
By another measure, they had finished just in time. A few weeks before they completed the final run, Thrun had learned their cover was blown. John Markoff, the New York Times tech reporter who had been in Stanley when the Grand Challenge champion crashed into a bush, had gotten a tip from the friend of a Google safety driver that the company was testing robot cars on public roads. Knowing a story was coming out no matter what, Thrun tried to ensure it made Google seem innovative, not reckless. He invited Markoff to climb into one of Chauffeur’s Priuses for an autonomous ride down the freeway and through Mountain View, with Urmson behind the wheel. While the team made clear that it had years of work to do before it could commercialize the tech, and still struggled with things like traffic cops’ hand gestures, its system impressed the reporter. “The Google research program using artificial intelligence to revolutionize the automobile is proof that the company’s ambitions reach beyond the search engine business,” he wrote in the October 9, 2010, article that introduced the world to Google’s self-driving car.