AUTONOMY

A NEW CHALLENGE

I begin with what is perhaps the only property that makes AI special. This is autonomy. We’re giving machines the ability to act in our world somewhat independently. And these actions can have an impact on humans. Not surprisingly, autonomy raises a whole host of interesting ethical challenges.

New technologies often introduce ethical challenges. And you could argue that autonomy is the one new ethical challenge that AI poses. All the other challenges that AI introduces, such as bias or invasion of our privacy, are ones we have faced before. For example, we have been trying to tackle racial bias for decades. AI may have put the problem on steroids, but it is an old problem. Similarly, many governments have been encroaching on our personal privacy for decades. AI may have put this problem on steroids too, but it is not a new problem.

Autonomy, on the other hand, is an entirely novel problem. We’ve never had machines before that could make decisions independently of their human masters. Previously, machines only did what we decided they should do. In some sense, machines in the past were only ever our servants. But we’ll soon have machines that make many of their own decisions. Indeed, anyone who owns a Tesla car already has one such machine.

Such autonomy introduces some very difficult new ethical questions. Who is accountable for the actions of an autonomous AI? What limits should be placed on an autonomous AI? And what happens if an autonomous AI harms or kills a person, purposefully or accidentally?

THE RUBBER HITS THE ROAD

The development of self-driving1 or autonomous cars is one place where we have seen some of the most detailed discussions around AI and ethics. This is unsurprising. A self-driving car is, in fact, a robot. And we don’t build many other robots that can travel at over 150 kilometres per hour.

At present, there are approximately 3 million robots working in factories around the world, and another 30 million or so in people’s homes. The total number of robots on the planet is thus slightly higher than the population of Australia. But robots will shortly outnumber all humans, and many of these will be self-driving cars.

A car might not look like a robot, but it is. A robot is a machine that can sense, reason and act. A self-driving car can sense the road and other users on the road. It then reasons about where the car needs to go. And finally it acts, following the road and avoiding obstacles. You can simply sit in the car and say, ‘Take me home.’ And the car will autonomously do the rest.

Self-driving cars will have profound consequences for our lives. One of the most obvious is improved safety. One million people around the world die in road traffic accidents every year. Over a thousand of these deaths are in Australia. This is a billion-dollar problem in Australia alone, as each fatal accident costs around $1 million to clear up. And that’s not counting the human cost.

Globally, road traffic accidents are one of the top ten major causes of death. In Australia, if you survive past your first birthday, it is the leading cause of death for the rest of your childhood. And almost all road traffic accidents are caused by human error, not mechanical failure. When we eliminate human drivers from the equation, cars will be much safer.

Computers aren’t going to drive tired, or drunk. They won’t text while driving, or make any of the other mistakes humans make. They’re going to be laser-focused on driving. They’ll calculate stopping distances exactly. They’ll be looking in all directions simultaneously. They’ll devote their vast computational resources to ensuring that no accidents occur.

In 2050, we’ll look back at 2000 and marvel at how things were. It’ll seem like the Wild West, with people driving cars and having accidents all the time. Often, we underestimate the rate of change. Wind back the clock to 1950 and the roads of Sydney looked much as they do today: full of cars, buses and the odd tram. But go back to 1900 and things were very different. It was mostly horses and carts. The world can become almost unrecognisable in less than 50 years.

By 2050, most of us won’t drive. Indeed, most of us won’t be able to drive. Young people won’t have bothered to learn to drive. They will go everywhere in self-driving Ubers, which will be as cheap as buses.2 And old people like you and me will go to renew our driving licences only to discover we haven’t had enough practice recently. Driving will be one of those activities, like subtraction or map reading, that humans used to do but that are now done almost exclusively by computers.

Driving in 2050 will be much like horseriding today. It used to be that most people could ride a horse – indeed, it was one of the major forms of transport. Horseriding has now, of course, become a pastime of the rich. Driving a car will go the same way by 2050. It will be an expensive hobby, reserved for special places like racetracks and tourist destinations.

THE UPSIDE

Self-driving cars won’t just make driving safer – they’ll have a host of other positive benefits. There are multiple groups who will profit – such as the elderly. My father recently and wisely decided he was no longer able to drive. But once we have self-driving cars, he’ll get back that mobility he has lost. Another group that will profit are those with disabilities, who will be given the mobility the rest of us take for granted. Similarly, autonomous cars will benefit the young. I’m looking forward to no longer being a taxi driver for my daughter – I’ll let our family car take over the responsibility of ferrying her around. And it will be far less worrying for me to know that a self-driving car is bringing her home late at night from a party, rather than one of her newly licensed friends.

Few inventions of the twentieth century have affected our lives as much as the automobile. It has shaped the very landscape of our cities. It has shaped where we live and work. It gave us mass production. The multistorey car park. Traffic lights and roundabouts. Cars take us to work during the week. To leisure activities at the weekend. And out to the country in the holidays.

The self-driving car will likely redefine the modern city. No longer will the commute to work in the morning and to home at night be ‘wasted’ time. We can use this time in our self-driving cars to clear our emails, to video-conference with our colleagues, even to catch up on sleep. Living in the suburbs or the country will become more attractive. This will be a good thing, as many of us will no longer be able to afford the ever-inflating property prices of the city centre.

It’s easy to describe the primary effect of self-driving cars: we won’t be driving anymore. But the secondary effects may be even more interesting. A continuation of the real estate boom in the country kickstarted by the COVID-19 pandemic? Inner cities as places of entertainment rather than of office work? Cars as offices? Might we stop buying cars all together, and just purchase time on some super-Uber self-driving car-sharing service?

Personally, I hate driving. I find it a waste of time. I can’t wait to hand over control to a computer, and get back the hours I now spend driving. My only concern is when I’ll be able to do so safely.

THE DOWNSIDE

In May 2016, 40-year-old Joshua Brown became the first person killed by a self-driving car. His Tesla Model S was driving autonomously down a highway near Williston, Florida, when an 18-wheel truck pulling a trailer full of blueberries turned across the car’s path. It was the middle of a bright spring day.

Unfortunately, the radar on the Tesla likely confused the high-sided vehicle for an overhead sign. And the cameras likely confused the white trailer for the sky. As a result, the car didn’t see the truck, did not brake, and drove into the 53-foot-long refrigerated truck at full speed. Actually, it was at more than full speed. The Model S was driving 9 miles per hour faster than the road’s speed limit of 65 miles per hour. You may be surprised to know that Tesla’s ‘Autopilot’ lets you set the car’s speed significantly above the speed limit.

As the two vehicles collided, the Tesla passed underneath the truck, with the windscreen of the Tesla hitting the bottom of the trailer. The top of the car was torn off by the force of the collision. The Tesla ran on and into a power pole. Joshua Brown died instantly from blunt force trauma to the head.

By many accounts, Joshua Brown was a technophile, an early adopter of new technologies. But like many of us, he appears to have placed a little too much faith in the capabilities of such new technologies. A month earlier, a video of his Tesla on Autopilot avoiding a collision with another truck caught Elon Musk’s attention. Joshua Brown tweeted ecstatically:

@elonmusk noticed my video! With so much testing/driving/talking about it to so many people I’m in 7th heaven!

Joshua Brown’s hands were on the wheel for only 25 seconds of the 37 minutes of his final journey. The Autopilot system warned him seven times before the fatal crash to place his hands back on the wheel. And seven times he removed his hands from the wheel. According to the Associated Press, the truck driver involved in the accident reported that Brown was actually watching a Harry Potter movie at the time of the crash. The police recovered a portable DVD player from the car.

In fact, Joshua Brown might not have been the first person killed by a self-driving car. Four months earlier, Gao Yaning, aged 23, died when the Tesla Model S that he was in drove into a road sweeper on a highway 300 miles south of Beijing in January 2016. However, that crash caused so much damage that Tesla claimed it could not determine if the Autopilot was engaged or not. There have been several more fatal accidents involving self-driving cars since then.

It was inevitable that a self-driving car would eventually kill an innocent pedestrian or cyclist. I made such a prediction at the end of 2016 in a list of AI trends for the coming year.3 Sadly, it took just over a year for my prediction to come doubly true. In Tempe, Arizona, in March 2018, a self-driving Uber test vehicle struck and killed Elaine Herzberg, a pedestrian pushing her bicycle across the road.

There were both technical and human reasons for this fatal accident. Uber’s self-driving system sensed the woman nearly six seconds before the impact. But the system failed to classify her as a pedestrian. She was crossing at a location without a pedestrian crossing, and the system had been set to ignore jaywalkers as it was giving too many false positives. The software also kept changing its classification of her – was she a vehicle? A bicycle? An unknown object? – preventing the car from braking or steering away.

When the self-driving car finally sounded the alarm to instruct Uber’s safety driver to intervene, she had only a fraction of a second in which to react. This is where human factors came into play. It didn’t help that when the alarm finally went off, the safety driver was watching an episode of The Voice on her mobile phone. She was subsequently charged with homicide and is awaiting trial.

The National Transportation Safety Board investigators looking into the accident were highly critical of Uber. They determined that the Arizona testing program lacked a formal safety plan, full-time safety staff and adequate operating procedures. Uber had also reduced its test drivers from two to one per vehicle just five months before the accident.

On the day of the Uber accident, the 3000 people who were killed in other road traffic accidents – those involving human-driven cars – didn’t make headlines around the world. Nor did the thousands of safe kilometres that other self-driving cars drove that day make the headlines. But then neither did the report that an Uber engineer sent to Eric Meyhofer, the leader of Uber’s self-driving car project. Less than a week before the accident, the engineer’s report warned about serious safety problems at Uber. ‘We shouldn’t be hitting things every 15,000 miles,’ the engineer wrote.

Except it wasn’t just a ‘thing’ that Uber was about to hit in March 2018. It was Elaine Herzberg.

HIGH STAKES

The Uber accident in Arizona brought home to me how much was in play in the development of self-driving cars. Actually, it wasn’t the fatal accident itself that made me realise how much was in play, but a spooky incident that took place on the night that news of the accident broke.

I was in a taxi going to a TV studio in Sydney to be interviewed on the national nightly news about the accident when my mobile phone rang. It wasn’t a number I recognised, but I picked it up just in case it was the producer of the program I was about to go on. The caller identified himself as the CEO of Volvo Australia.

Uber’s self-driving car was developed from Volvo’s XC90 sports utility vehicle. Uber had taken Volvo’s hardware platform and added its own software. Indeed, Volvo’s in-house semi-autonomous emergency braking system, which might have prevented the accident, had been switched off by Uber. What was very relevant to this unexpected phone call was that, in 2015, Volvo’s global CEO, Håkan Samuelsson, had declared that his company would accept full liability whenever one of its cars was in autonomous mode.

My caller wanted me to understand that Volvo was accepting no liability whatsoever for the Arizona accident. It was their car. But Uber had changed the software, so Volvo wanted me to know that it was entirely Uber’s responsibility.

I was impressed. How did Volvo know I was about to go on TV to discuss the accident? How did they get my mobile phone number? And, given it was their CEO, and not some random PR person, how much must Volvo care?

Around 75 million new cars are sold around the globe every year. That’s two cars sold every second. This adds up to well over a trillion dollars of sales annually. Second-hand cars roughly double that again.

The major car companies are worth significant amounts of money. In August 2021, Toyota had a market capitalisation of around $230 billion, the Volkswagen Group was worth $143 billion, Mercedes-Benz $87 billion, General Motors $70 billion, BMW $59 billion, and Honda a little less again at around $52 billion.

However, there is a new kid on the block that has upset this 100-year-old market. Tesla. In 2019, Tesla manufactured 367,500 cars, triple what the company made in 2017. This was still well behind Toyota, which built over 10 million cars in 2019, some 30 times more. Nevertheless, Tesla is already one of the ten most valuable US companies. In August 2021, it was worth over $670 billion, about triple the market cap of Toyota.4 In fact, Tesla has about a third of the market cap of the whole automotive sector.

The competition doesn’t end with cornering the market in building self-driving cars. The race is also on to capture markets around their deployment. Take ride-sharing services like Uber. Perhaps the only way that Uber can turn a profit is if they eliminate the most expensive part of their business: the human sitting in the driver’s seat. In this way, all the revenue can come back to Uber. Not surprisingly, Uber is at the forefront of developing self-driving cars.

In August 2021, Uber had a market cap of around $75 billion, on annual revenues of around $14 billion. In addition, there are competitors like Lyft, which has a market cap of around $15 billion, and the Chinese ride-sharing giant DiDi, which is a private company but worth around the same as Uber.

No one knows who the winner in the race to build and deploy self-driving cars will be. Those in Silicon Valley see it largely as a software problem. The winners could therefore be Tesla, Apple, Uber or Waymo (Alphabet), just as much as they could be the incumbents: Toyota, Volvo and General Motors. Whoever wins, it is going to be a very valuable pie. That just might help explain the unexpected phone call I received from the CEO of Volvo Australia while en route to my TV interview.

HOW SELF-DRIVING CARS DRIVE

To understand some of the ethical issues around self-driving cars, as well as the cause of some of these accidents, it helps to have a bit of an understanding about how self-driving cars actually work. You can’t yet buy a Level 5, fully autonomous self-driving car that can drive safely on public roads. But we have a good idea of the technology that will go into such a vehicle.

Level 5 means the car requires no human interaction. The vehicle should be able to steer, accelerate, brake and monitor road conditions without needing the driver to pay any attention to the car’s functions whatsoever. Level 5 is the highest level of automation defined by the Society of Automotive Engineers, and has been adopted by the US Department of Transportation.5

Companies like Waymo and Tesla have tested cars rated at Level 5 on public roads in multiple states in the United States. Level 5 vehicles have also been deployed in places such as airports and parking lots. In fact, Australia leads the world in this, with fully autonomous vehicles trundling around many mines and ports.

Looking ahead, you can probably expect to come across autonomous vehicles first in geographically constrained spaces, rather than on the open road. This might be a regulated space like a mine. Or it might be the high-speed lane of the motorway, where only autonomous cars that can safely platoon together are allowed.

As to when Level 5 cars will be widely available for sale, it’s hard to say. Elon Musk would like you to believe this time might be measured in months. Others have suggested it might be decades. But no one is suggesting it will be never. It’s not too soon, therefore, to start planning for their existence.

First and foremost, self-driving cars are very careful map followers. They have high-precision maps, accurate to the centimetre. And they use GPS and other sensors like LIDAR to locate the car precisely within those maps. But you can’t drive by just following a map. You need also to sense the world. You need to identify other vehicles and users of the road. You need to avoid obstacles, and cope with places where the map is wrong. Self-driving cars therefore have a range of different sensors with which to observe the world.

One of the primary sensors in a self-driving car is a camera. Just like with human drivers, vision is a good way to observe the world. Cameras are cheap and so can be scattered around the car. Indeed, unlike a human driver, cameras can provide a continuous 360-degree view of the world. However, they have a number of limitations. Poor weather conditions such as rain, fog or snow will prevent cameras from seeing obstacles clearly. Additionally, there are often situations, like in low light, where the images from these cameras simply aren’t adequate for a computer to decide what to do.

Self-driving cars therefore have a number of other sensors. For example, both short-range and long-range radar sensors are deployed all around the car. Radar systems, unlike cameras, are active sensors and so can work at night, or in weather conditions like rain or fog. Long-range radar sensors facilitate distance control and automatic braking. Short-range radar sensors, on the other hand, are used for actions like blind spot monitoring. For even shorter distances, to enable activities like parking, ultrasonic sensors are used.

One of the most useful sensors is LIDAR. This is similar to radar except it uses light produced by a laser instead of radio waves to illuminate the world around the car. LIDAR measures the distance to objects accurately using time of flight, giving a detailed 360-degree cloud of the obstacles surrounding the car. If you’ve ever seen one of Google’s self-driving cars, LIDAR is the rotating ice-cream bucket on the roof.

LIDAR used to be very expensive, costing tens of thousands of dollars. But prices have been plummeting, and LIDAR now costs in the hundreds of dollars. Indeed, LIDAR is so small and cheap now that you’ll find it on the latest iPads and iPhones. But while you’ll find LIDAR on almost every self-driving car, you won’t find it on a Tesla.

Elon Musk has said, ‘LIDAR is a fool’s errand. Anyone relying on LIDAR is doomed. Doomed! [They are] expensive sensors that are unnecessary. It’s like having a whole bunch of expensive appendices. Like, one appendix is bad, well now you have a whole bunch of them, it’s ridiculous, you’ll see.’6

The funny thing is that no other company trying to build autonomous cars besides Tesla is trying to do it without LIDAR. Not using LIDAR is tying your own hands behind your back. The main reason Tesla seems to have gone down this route is that LIDAR was too expensive to put into a Tesla a few years ago. And Tesla wanted to sell you a car then that, after a software upgrade in a few years’ time, would provide Level 5 autonomy. Indeed, Tesla is already selling this future software upgrade.

A LIDAR system would very likely have detected the truck crossing the road that killed Joshua Brown in Florida in 2016. Its laser beam would have reflected off the flat side of the trailer as it crossed the path of the car. This would have clearly indicated that the road ahead was blocked. And, instead of driving on blindly, the car would have braked automatically.

I have no desire to repeat Joshua Brown’s mistake. I want any self-driving car I trust with my life to be literally laser-focused on any obstacles. I won’t therefore be trusting a Level 5 self-driving car that lacks LIDAR. And neither should you.

MAGNIFICENT MACHINES

How can it be that companies building self-driving cars get to choose whether to include a safety device like LIDAR? Why is there such limited government oversight of how the industry develops? How can the public be confident that corners are not being cut in the race to market? Can we learn any lessons from other industries, such as aviation or pharmaceuticals?

Suppose a drug company was testing a new product on the public. And imagine that the company didn’t have to get approval from an independent ethics board for the trial. That the company wasn’t getting informed consent from the public. That the drug was experimental, and the company was expecting some fatal accidents. And that it had, in fact, already killed several members of the public, including innocent bystanders.

If I told you all that, you would be outraged. You’d tell me that surely there are laws preventing this sort of thing. And you’d be right: there are laws preventing such harm from taking place – in the pharmaceutical industry. But this is essentially what is happening today in the development of self-driving cars. Technology companies are testing their prototype self-driving cars on the public roads. They don’t have to get much in the way of ethics approval for the trials. They aren’t getting informed consent from the public. And they’ve already killed several people, including innocent bystanders.

How can there not be a government body with careful oversight of the development of self-driving cars? We could perhaps take a lesson from history. One hundred years ago, we developed a new form of transport. And despite the inherent risks, we made it the safest way to get from A to B, by orders of magnitude. This success story is the story of aviation.

In the early days of the aviation industry, those magnificent flying planes were incredibly dangerous. They fell out of the sky with distressing regularity. In 1908, four months after the very first passenger flight, Thomas Selfridge became the industry’s first fatality when the Wright Model A flyer crashed, also seriously injuring Orville Wright himself. But somehow the aviation industry turned this around, so that now you’re more likely to be killed in a car going to the airport than during the flight itself.

To do this, we set up independent bodies, such as the Australian Transport Safety Bureau, to investigate the cause of accidents, and we mandated that they share those findings with all industry players. We also set up bodies like the Civil Aviation Safety Authority to license pilots, ground crew, aircraft and airfield operators. In this way, airplane accidents don’t get repeated. Aircraft safety is like a ratchet: planes only ever get safer.

Contrast this with the competitive race to develop self-driving cars. There is no sharing of information within the industry. Indeed, the only information that the companies developing self-driving cars share is the information they steal from each other. In August 2020, former Waymo engineer Anthony Levandowski was sentenced to 18 months in prison for stealing details of Waymo’s self-driving car, which he took with him when he moved to Uber.

TROLLEY PROBLEMS

The most discussed fatal crash involving a self-driving car is – ironically – one that has never happened. Imagine a self-driving car with two people on board that drives around a corner. Crossing the road a short distance ahead is an old lady. There is no time for the car to stop, so the onboard computer has to make a decision. Does the car run over the old lady, surely killing her? Or does it run off the road and into a brick wall, surely killing both its own occupants?

This moral dilemma is known as a ‘trolley problem’. It was dreamed up by the English philosopher Philippa Foot in 1967. It’s called a trolley problem since the original formulation imagined a runaway trolley heading down a railway track. Five men are working on this track, and are all certain to die when the trolley reaches them. However, it’s possible for you to switch the trolley’s path to an alternative spur of track, saving all five lives. Unfortunately, one man is working on this spur, and he will be killed if the switch is made. Do you switch the trolley to save the lives of the five men, sacrificing the man on the spur? In surveys, 90 per cent of people switch the trolley, sacrificing the man on the spur but saving five lives.

There are many variants of the trolley problem that explore people’s moral decision-making. What if the trolley goes under a footbridge, and there is a fat man on the footbridge who you can push off and into the path of the trolley? The fat man will surely stop the trolley but be killed by the collision. Should you kill the fat man in order to save those five lives? In surveys, some 90 per cent of people would not. Such premeditated murder seems different to throwing the switch in the first example, even though the outcomes are the same: one person dead instead of five.

Another variant of the trolley problem has five people waiting in hospital for transplants of a heart, lung, liver and kidney. Each person will die if they don’t have a transplant in the very near future. A fit young person walks into the hospital. You could harvest this young person’s organs and save the lives of five people. Such cold-blooded murder seems very different to many people, even if the choice is the same: to kill one person to save five lives.

There has been so much discussion about trolley problems like this that philosophers now playfully talk about ‘trolleyology’, the study of ethical dilemmas. This is an academic discipline that brings together philosophy, behavioural psychology and artificial intelligence.

Trolley problems are not, in one sense, new. Every time you’ve driven a car, you might have had to face such a trolley problem at any moment. But what is new is that we now have to code the answer for how to behave in such a situation in advance. If you study driving codes around the world, none of them specifies precisely what to do in such a circumstance. You will have a split second to decide. If you survive, and you acted unwisely, you may face manslaughter charges. We haven’t had to worry previously about how to codify what to do in such a life-or-death situation.

In June 2017, the German Federal Ministry of Transport and Digital Infrastructure released a report containing Germany’s ethical guidelines for self-driving cars.7 The report wishfully recommended that ‘the technology must be designed in such a way that critical situations [like the trolley problem] do not arise in the first place’. But since outlawing trolley problems is not actually possible, the report specifies constraints on how autonomous vehicles should be programmed in such situations:

In the event of unavoidable accident situations, any distinction based on personal features (age, gender, physical or mental constitution) is strictly prohibited. It is also prohibited to offset victims against one another. General programming to reduce the number of personal injuries may be justifiable. Those parties involved in the generation of mobility risks must not sacrifice non-involved parties.

There are three fundamental difficulties with the trolley problem. First, 50 years after its creation, we’re still struggling to provide a definitive answer to the problem. This shouldn’t be surprising. The trolley problem is a moral dilemma – it wasn’t designed to have a definitive solution. Dilemmas are, by their very nature, ambiguous, even unanswerable.

The trolley problem permits us to explore the moral tension between deontological reasoning (where we judge the nature of an action rather than its consequences) and consequentialism (where we judge an action by its consequences). It is wrong to kill versus it is good to save lives. Somehow I doubt that a bunch of computer programmers are going to solve a moral dilemma like this, which has resisted solution by very smart philosophers over decades of argument.

This tension between deontology and consequentialism is at the heart of most variants of the trolley problem. Yet perhaps the dilemma most troubling implication is not the existence of this tension, but the fact that, depending on how the trolley problem is framed, people have wildly different views about the right course of action. How can we hope to code moral behaviours when humans can’t agree?

The ethical guidelines of the German Federal Ministry of Transport and Digital Infrastructure reflect this tension. They try to define the dilemma away, mandating that such situations should not occur, so they do not require resolution. While trolley problems are rare, we cannot simply prohibit them by means of some Teutonic precision.

I can still picture the London street on a bright summer morning in 1983 when I faced my own personal trolley problem. I had recently passed my test and was driving my red Mini to work. A car pulled out of a side street to the left of my path. I had a split second to decide between hitting the car or swerving across the road and onto a pedestrian crossing, where a woman and child were crossing. I am not sure if I wisely chose to hit the car or whether I just froze. But in any case there was an almighty bang. The two cars were badly damaged, but fortunately no one was greatly hurt.

The second issue with the trolley problem is that it wasn’t dreamed up as a practical moral question. It certainly wasn’t intended to describe a real moral dilemma that might be encountered by an autonomous vehicle. It was invented for an entirely different purpose.

The trolley problem was proposed by Philippa Foot as a less controversial way of discussing the moral issues around abortion. When is it reasonable, for example, to kill an unborn child to save the life of the mother? Fifty years later, we still don’t have definitive answers to such questions. Abortion was made legal in the United Kingdom in 1967. But it was only decriminalised in New South Wales in October 2019, having been part of the criminal code for 119 years. How can we expect AI researchers and automotive engineers to code matters of life and death when our broader society has struggled with these issues for decades?

The third and very practical difficulty with the trolley problem is that it is irrelevant to those who program self-driving cars. I know quite a few people who program self-driving cars. If you ask them about the part of the computer code that decides the trolley problems, they look at you blankly. There is no such code. The top-level control loop of a self-driving car is simply to ‘drive on the green road’. Traditionally, programmers of self-driving cars have painted the road in front of the car in green to indicate where there are no obstacles and it is safe to drive.8 And if there’s no green road, the car is programmed simply to brake as hard as possible.

Self-driving cars today, and likely for a very long time in the future, simply don’t understand the world well enough to solve trolley problems. Self-driving cars are not in a position to trade off lives here for lives over there. The car’s perception and understanding of the world is inadequate to make the subtle distinctions discussed in trolley problems.

A self-driving car is like a nervous human learning to drive a car down the road, and saying out loud: Don’t crash . . . Don’t crash . . . Stay on the green road . . . Don’t crash . . . Stay on the green road . . . Oh no, I’m going to crash – I’d better brake . . . Brake . . . Brake . . . If more people appreciated this simple reality, there would be fewer Joshua Browns putting too much trust in the technology and paying with their lives.

MORAL MACHINES

The Media Lab at the Massachusetts Institute of Technology (MIT) is the ‘show pony’ of the technology world. Ever since it was founded by the charismatic Nicholas Negroponte in 1985, the Media Lab has been famous for its flashy demos. In 2019, it came under fire for secretly accepting donations from convicted child sex offender Jeffrey Epstein. This was after MIT had officially declared that the disgraced financier was disqualified from being a donor.

Negroponte is perhaps best known for the inspirational but poorly executed One Laptop per Child project, launched at the World Economic Forum at Davos in 2005. Negroponte wanted to put $100 laptops in the hands of hundreds of millions of children across the developing world. The project attracted a lot of publicity, but ultimately failed to deliver.

Actually, that’s a good description of many Media Lab projects. Another is the Moral Machine. You can check it out at moralmachine.net. It’s a platform for crowdsourcing human perspectives on the moral decisions made by machines, such as self-driving cars. Using an interface not too different from Tinder’s, humans can vote on how self-driving cars should behave in a particular trolley problem.

Continue straight into some road works containing a concrete block and kill the two elderly occupants of the car OR swerve across the road and only kill a young child on a pedestrian crossing. How do you vote?

To date, millions of people from almost every country in the world have voted on over 40 million moral choices at moralmachine.net. The goal of the platform is to collect data to ‘provide a quantitative picture of the public’s trust in intelligent machines, and of their expectations of how they should behave’.⁹ Is it that simple? Can we simply program self-driving cars using data from the aptly named ‘moral machine’?

There are many reasons to be sceptical. First, we humans often say one thing but do another. We might say that we want to lose weight, but we might still eat a plate full of delicious cream doughnuts. What we tell the moral machine may bear little resemblance to how we might actually behave in a real-world life-or-death situation. Sitting in front of a computer and swiping left or right is nothing like gripping the steering wheel of a car and taking your life in your own sweaty hands.

The second reason to be sceptical of the moral machine is that even if we say what we might actually do, there’s a lot we actually do that we shouldn’t. We are only human, after all, and our actions are not always the right ones. Just because I ate the cream doughnut from my autonomous self-stocking fridge doesn’t mean I want the fridge to order me more cream doughnuts. In fact, I want my autonomous fridge to do the opposite. Please stop ordering more fattening temptation.

The third reason to be sceptical is that there’s little rigour behind the experiment. But despite this, the inventors of the moral machine have made some bold claims. For example, they say that people in Western countries are more likely to sacrifice the fat man by pushing him off the footbridge than people from Eastern countries, and that this likely reflects differing attitudes to the value of human life in these different societies.10 Such claims are problematic, as the people using the moral machine are not demographically balanced. They are a self-selecting group of internet users. Unsurprisingly, they are mostly young, college-educated men. In addition, there’s no attempt to ensure that their answers are reasonable. To learn more about the moral machine experiment, I took several of its surveys. Each time, I perversely chose to kill the most people possible. The moral machine never once threw me out.

The fourth and final reason to be sceptical of the moral machine is that moral decisions are not some blurred average of what people tend to do. Moral decisions may be difficult and rare. Morality changes. There are many decisions we used to make that we no longer think are moral. We used to deny women the vote. We used to enslave people. We no longer believe either of those are morally acceptable.

Like the trolley problem upon which it is based, the moral machine has attracted a lot of publicity. It is exactly the sort of project you might expect to come out of the Media Lab. But it’s much less clear to me whether the moral machine has actually advanced the challenge of ensuring that autonomous machines act in ethical ways.

KILLER ROBOTS

Self-driving cars are not designed to kill. In fact, they are designed to do the opposite – to save lives. But when things go wrong, they may accidentally kill people. There are, however, other autonomous machines entering our lives which are designed expressly to kill: these are ‘lethal autonomous weapons’ – or, as the media like to call them, ‘killer robots’.

The world faces a critical choice about this application of autonomy on the battlefield. This is not due to the growing political movement against lethal autonomous weapons. Thirty jurisdictions have called on the United Nations to ban such weapons pre-emptively. I want to list them by name to recognise their moral leadership: Algeria, Argentina, Austria, Bolivia, Brazil, Chile, China, Colombia, Costa Rica, Cuba, Djibouti, Ecuador, Egypt, El Salvador, Ghana, Guatemala, the Holy See, Iraq, Jordan, Mexico, Morocco, Namibia, Nicaragua, Pakistan, Panama, Peru, the State of Palestine, Uganda, Venezuela and Zimbabwe.

Both the African Union and the European Parliament have come out in support of such a ban. In March 2019, the German foreign minister, Heiko Maas, called for international cooperation on regulating autonomous weapons. And in the same week that Maas called for action, Japan gave its backing to global efforts at the United Nations to regulate the development of lethal autonomous weapons.

At the end of 2018, the United Nations’ secretary-general, António Guterres, addressing the General Assembly, offered a stark warning:

The weaponization of artificial intelligence is a growing concern.

The prospect of weapons that can select and attack a target on their own raises multiple alarms – and could trigger new arms races.

Diminished oversight of weapons has implications for our efforts to contain threats, to prevent escalation and to adhere to international humanitarian and human rights law.

Let’s call it as it is. The prospect of machines with the discretion and power to take human life is morally repugnant.

However, it’s not this growing political and moral concern that underlines the critical choice we face today around killer robots. Nor is it the growing movement within civil society against such weapons. The Campaign to Stop Killer Robots, for instance, now numbers over 100 non-governmental organisations, such as Human Rights Watch which are vigorously calling for regulation. But it’s not the pressure of such NGOs to take action that has brought us to this vital juncture.

Nor is it the growing concern of the public around killer robots. A recent international IPSOS poll11 showed that opposition to fully autonomous weapons has increased 10 per cent in the last two years as understanding of the issues grows. Six out of every ten people in 26 countries strongly opposed the use of autonomous weapons. In Spain, for example, two-thirds of those polled were strongly opposed, while not even one in five people supported their use. The levels were similar in France, Germany and several other European countries.

No, the reason we face a critical choice today about the future of AI in warfare is that the technology to build autonomous weapons is ready to leave the research lab and be developed and sold by arms manufacturers around the world.

In March 2019, for instance, the Royal Australian Air Force announced a partnership with Boeing to develop an unmanned air combat vehicle, a loyal ‘wingman’ that would take air combat to the next step of lethality. The project is part of Australia’s $35-million Trusted Autonomous Systems program, which aims to deliver trustworthy AI into the Australian military. In the same week, the US Army announced ATLAS, the Advanced Targeting and Lethality Automated System, which will be a robot tank. The US Navy, too, announced that its first fully autonomous ship, Sea Hunter, had made a record-breaking voyage from Hawaii to the Californian coast without human intervention.

Unfortunately, the world will be a much worse place if, in a decade, militaries around the planet are regularly using such lethal autonomous weapons systems – also known as LAWS – and if there are no laws regulating LAWS. This, then, is the critical choice we face today. Do we let militaries around the world build such technologies without constraints?

The media like to use the term ‘killer robot’ rather than a wordier expression such as lethal or fully autonomous weapon. The problem is that ‘killer robot’ conjures up an image of Hollywood’s Terminator. And it is not something like the Terminator that worries me or thousands of my colleagues working in AI. It’s the much simpler technologies that we see being developed today.

Take the Predator drone. This is a semi-autonomous weapon. It can fly itself for much of the time. However, there is still a soldier, typically in a container in Nevada, who is in overall control of the drone. And, importantly, it is still a soldier who makes the decision to fire one of its deadly Hellfire missiles. But it is a small technological step to replace that soldier with a computer, and empower that computer to identify, track and target. In fact, it is technically possible today.

At the start of 2020, the Turkish military deployed the Kargu quad-copter drone that was developed by the Turkish arms company STM. The drone is believed to be able to autonomously swarm, identify targets using computer-vision and face-recognition algorithms, and destroy those targets on the ground by means of a kamikaze attack.

Once such autonomous weapons are operational, there will be an arms race to develop more and more sophisticated versions. Indeed, we can already see the beginnings of this arms race. In every theatre of war – in the air, on land, and on and under the sea – there are prototypes of autonomous weapons being developed. This will be a terrible transformation of warfare.

But it is not inevitable. In fact, we get to choose whether we go down this road. For several years now, I and thousands of my colleagues, other researchers in the artificial intelligence and robotics research communities, have been warning of these dangerous developments. We’ve been joined by founders of AI and robotics companies, Nobel laureates, church leaders, politicians and many members of the public.

Strategically, autonomous weapons are a military dream. They let a military scale their operations unhindered by manpower constraints. One programmer can command hundreds of autonomous weapons. This will industrialise warfare. Autonomous weapons will greatly increase strategic options. They will take humans out of harm’s way, making the riskiest of missions more feasible. You could call it War 3.0.

There are many reasons, however, why the military’s dream of lethal autonomous weapons systems will turn into a nightmare. First and foremost, there is a strong moral argument against killer robots. We give up an essential part of our humanity if we hand over military decisions to a machine. Machines have no emotions, compassion or empathy. How, then, can they be fit to decide who lives and who dies?

Beyond the moral arguments, there are many technical and legal reasons to be concerned about killer robots. In the view of my colleague Stuart Russell, co-author of the definitive textbook on AI, one of the strongest arguments is that they will revolutionise warfare. Autonomous weapons will be weapons of immense and targeted destruction. Previously, if you wanted to do harm, you had to have an army of soldiers to wage war. You had to persuade this army to follow your orders. You had to train them, feed them and pay them. Now a single programmer could control hundreds of weapons.

Lethal autonomous weapons are more troubling, in some respects, than nuclear weapons. To build a nuclear bomb requires technical expertise – you need skilled physicists and engineers. You need the resources of a nation-state, and access to fissile material. Nuclear weapons have not, as a result, proliferated greatly. Autonomous weapons require none of this. You’ll just need a 3D printer and some sophisticated code that is easily copied.

If you’re not yet convinced that autonomous weapons might pose a bigger threat than nuclear weapons, then I have some more bad news for you. Russia has announced plans to build Poseidon, an autonomous nuclear-powered and nuclear-tipped underwater submarine.12 Can you think of anything more horrifying than an algorithm that can decide to start a nuclear war?

But even non-nuclear autonomous weapons will be a dreadful scourge. Imagine how terrifying it will be to be chased by a swarm of autonomous kamikaze drones. Autonomous weapons will likely fall into the hands of terrorists and rogue states, who will have no qualms about turning them on innocent civilians. They will be an ideal weapon with which to suppress a population. Unlike humans, they will not hesitate to commit atrocities, even genocide.

LAWS BANNING LAWS

You may be surprised to hear this, but not everyone is on board with the idea that the world would be a better place with a ban on killer robots. ‘Robots will be better at war than humans,’ they say. ‘They will follow their instructions to the letter. Let robot fight robot and keep humans out of it.’

Such arguments don’t stand up to much scrutiny, in my view. Nor do they stand up to the scrutiny of many of my colleagues in AI and robotics. Here are the five main objections I hear to banning killer robots – and why they’re misguided.

Objection 1: Robots will be more effective than humans.

They’ll be more efficient for sure. They won’t need to sleep. They won’t need time to rest and recover. They won’t need long training programs. They won’t mind extreme cold or heat. All in all, they’ll make ideal soldiers.

But they won’t be more effective. The leaked ‘Drone Papers’, published by the Intercept in 2015, recorded that nearly nine out of ten people killed by drone strikes weren’t the intended target.13 And this is when there was still a human in the loop, making the final life-or-death decision. The statistics will be even worse when we replace that human with a computer.

Killer robots will also be more efficient at killing us. Terrorists and rogue nations are sure to use them against us. It’s clear that if the weapons not banned, then there will be an arms race. It is not overblown to suggest that this will be the next great revolution in warfare, after the invention of gunpowder and nuclear bombs. The history of warfare is largely one of who can more efficiently kill the other side. This has typically not been a good thing for humankind.

Objection 2: Robots will be more ethical.

This is perhaps the most interesting argument. Indeed, there are even a few who claim that this moral argument requires us to develop autonomous weapons. In the terror of battle, humans have committed many atrocities. And robots can be built to follow precise rules. However, as this book shows, it’s fanciful to imagine we know how to build ethical robots. AI researchers have only just started to worry about how you could program a robot to behave ethically. It will take us many decades to work out how to deploy AI responsibly, especially in a high-stakes setting like the battlefield.

And even if we could, there’s no computer we know of that can’t be hacked to behave in ways that we don’t desire. Robots today cannot make the distinctions that the international rules of war require: to distinguish between combatant and civilian, to act proportionally and so on. Robot warfare is likely to be a lot more unpleasant than the wars we fight today.

Objection 3: Robots can just fight robots.

Replacing humans with robots in a dangerous place like the battlefield might seem like a good idea. However, it’s fanciful to suppose that we could just have robots fighting robots. Wars are fought in our towns and cities, and civilians are often caught in the crossfire, as we have sadly witnessed recently in Syria and elsewhere. There’s not some separate part of the world called ‘the battlefield’ where there are only robots.

Also, our opponents today are typically terrorists and rogue states. They are not going to sign up to a contest between robots. Indeed, there’s an argument that the terror unleashed remotely by drones has likely aggravated the many conflicts in which we find ourselves enmeshed currently. To resolve some of these difficult situations, we have to put not robots but boots on the ground.

Objection 4: Such robots already exist, and we need them.

There are some autonomous weapons deployed by militaries today – weapons systems like the Phalanx CIWS (close-in weapon system), which sits on many US, British and Australian naval ships. Or Israel’s Harpy weapon, a kamikaze drone that loiters for up to six hours over a battlefield and uses its anti-radar homing system to take out surface-to-air missile systems on the ground.

I am perfectly happy to concede that a technology like the autonomous Phalanx system is a good thing. But the Phalanx is a defensive system, and you don’t have time to get a human decision when defending yourself against an incoming supersonic missile. I and other AI researchers have called for offensive autonomous systems to be banned, especially those that target humans. An example of the latter is the autonomous Kargu drone, currently active on the Turkish–Syrian border. This uses the same facial-recognition algorithms as in your smartphone, with all of their errors, to identify and target people on the ground.

There’s no reason why we can’t ban a weapons system that already exists. Indeed, most bans, like those for chemical weapons or cluster munitions, have been for weapons systems that not only exist but have been used in war.

Objection 5: Weapon bans don’t work.

History would contradict the argument that weapon bans don’t work. The 1998 UN Protocol on Blinding Lasers has resulted in lasers that are designed to blind combatants permanently being kept off the battlefield. If you go to Syria today – or to any other war zones – you won’t find this weapon. And not a single arms company anywhere in the world will sell you one. You can’t uninvent the technology that supports blinding lasers, but there’s enough stigma associated with them that arms companies have stayed away.

I hope a similar stigma will be associated with autonomous weapons. It’s not possible to uninvent the technology, but we can put enough stigma in place that robots aren’t weaponised. Even a partially effective ban would likely be worth having. Anti-personnel mines still exist today despite the 1997 Ottawa Treaty. But 40 million such mines have been destroyed. This has made the world a safer place, and resulted in many fewer children losing a limb or their life.

*

AI and robotics can be used for many great purposes. Much of the same technology will be needed in autonomous cars as in autonomous drones. And autonomous cars are predicted to save 30,000 lives on the roads of the United States alone every year. They will make our roads, factories, mines and ports safer and more efficient. They will make our lives healthier, wealthier and happier. In the military setting, there are many good uses of AI. Robots can be used to clear minefields, take supplies through dangerous routes and shift mountains of signal intelligence. But they shouldn’t be used to kill.

We stand at a crossroads on this issue. I believe it needs to be seen as morally unacceptable for machines to decide who lives and who dies. If the nations of the world agree, we may be able to save ourselves and our children from this terrible future.

In July 2015, I helped organise an open letter to the United Nations calling for action that was signed by thousands of my colleagues. The letter was released at the start of the main international AI conference. Sadly, the concerns we raised in this letter have yet to be addressed. Indeed, they have only become more urgent. Here is the text of our open letter:

AUTONOMOUS WEAPONS: AN OPEN LETTER FROM AI & ROBOTICS RESEARCHERS

Autonomous weapons select and engage targets without human intervention. They might include, for example, armed quadcopters that can search for and eliminate people meeting certain pre-defined criteria, but do not include cruise missiles or remotely piloted drones for which humans make all targeting decisions. Artificial Intelligence (AI) technology has reached a point where the deployment of such systems is – practically if not legally – feasible within years, not decades, and the stakes are high: autonomous weapons have been described as the third revolution in warfare, after gunpowder and nuclear arms.

Many arguments have been made for and against autonomous weapons, for example that replacing human soldiers by machines is good by reducing casualties for the owner but bad by thereby lowering the threshold for going to battle. The key question for humanity today is whether to start a global AI arms race or to prevent it from starting. If any major military power pushes ahead with AI weapon development, a global arms race is virtually inevitable, and the endpoint of this technological trajectory is obvious: autonomous weapons will become the Kalashnikovs of tomorrow.

Unlike nuclear weapons, they require no costly or hard-to-obtain raw materials, so they will become ubiquitous and cheap for all significant military powers to mass-produce. It will only be a matter of time until they appear on the black market and in the hands of terrorists, dictators wishing to better control their populace, warlords wishing to perpetrate ethnic cleansing etc. Autonomous weapons are ideal for tasks such as assassinations, destabilising nations, subduing populations and selectively killing a particular ethnic group. We therefore believe that a military AI arms race would not be beneficial for humanity. There are many ways in which AI can make battlefields safer for humans, especially civilians, without creating new tools for killing people.

Just as most chemists and biologists have no interest in building chemical or biological weapons, most AI researchers have no interest in building AI weapons – and do not want others to tarnish their field by doing so, potentially creating a major public backlash against AI that curtails its future societal benefits. Indeed, chemists and biologists have broadly supported international agreements that have successfully prohibited chemical and biological weapons, just as most physicists supported the treaties banning space-based nuclear weapons and blinding laser weapons.

In summary, we believe that AI has great potential to benefit humanity in many ways, and that the goal of the field should be to do so. Starting a military AI arms race is a bad idea, and should be prevented by a ban on offensive autonomous weapons beyond meaningful human control.14

In 2020, five years after we wrote this letter, the United Nations is still discussing the idea of regulating killer robots. And the military AI arms race that we warned about has clearly begun.

THE RULES OF WAR

If the United Nations fails to prohibit killer robots in the near future, we will have to work out how to build robots that will follow the rules of war. From the outside, war might appear to be a rather lawless activity. A lot of people get killed in war, and killing people is generally not permitted in peacetime. But there are internationally agreed rules for fighting war. And these rules apply to robots as much as to people.

The rules of war distinguish between jus ad bellum – one’s right to go to war – and jus in bello, one’s rights while at war. To put it in plainer language, the rules of war distinguish between the conditions under which states may resort to war and, once states are legally at war, the way they conduct warfare. The two concepts are deliberately independent of each other. Jus ad bellum requires, for example, that war must be fought for a just cause, such as to save life or protect human rights. It also requires that war must be defensive and not aggressive, and that it must be declared by a competent authority such as a government. For the present, it is unlikely that machines are going to be declaring war by themselves. It is perhaps reasonable to suppose, therefore, that humans are still going to be the ones taking us to war. So I’ll put aside for now concerns about killer robots accidentally starting a war, and focus instead on jus in bello.

The rules governing the conduct of war seek to minimise suffering, and to protect all victims of armed conflict, especially non-combatants. The rules apply to both sides, irrespective of the reasons for the conflict or the justness of the causes for which they are fighting. If it were otherwise, the laws would be pretty useless, as each party would undoubtably claim to be the victim of aggression.

There are four main principles of jus in bello. We begin with the principle of humanity, which also goes under the name of the ‘Martens Clause’. This was introduced in the preamble to the 1899 Hague Convention by Friedrich Martens, a Russian delegate. It requires war to be fought according to the laws of humanity, and the dictates of the public conscience.

The Martens Clause is a somewhat vague principle, a catch-all that outlaws behaviours and weapons that the public might find repugnant. How, for instance, do we determine precisely the public conscience? The Martens Clause is often interpreted to prefer, for example, capturing an enemy over wounding them, and wounding over killing, and to prohibit weapons that cause excessive injury or pain.

The second principle of jus in bello is that of distinction. You must distinguish between the civilian population and combatants, and between civilian objects and military objectives. The only legitimate target is a military objective. It requires defenders to avoid placing military personnel or matériel in or near civilian objects, and attackers to use only those methods of assault that are discriminating in effect.

The third principle of jus in bello is that of proportionality. This prohibits attacks against military objectives which are expected to cause incidental loss of civilian life, injury to civilians or damage to civilian objects which would be excessive compared to the expected military advantage from that attack. This principle requires attackers to take precautions to minimise collateral damage, and to choose, where possible, objectives likely to cause the least danger to civilians and civilian objects.

The fourth and final principle of jus in bello is that of military necessity. This limits armed force to those actions that have legitimate military objectives. This means avoiding inflicting gratuitous injury on the enemy. The principle of necessity overlaps in part with the Martens Clause. Both take account of humanitarian concerns around the wounding of soldiers. And both prohibit weapons that cause unnecessary suffering.

In my view, lethal autonomous weapons today fail to uphold all four principles of jus in bello, the conduct of war. Consider, for example, the Martens Clause. The majority of the public are against the idea of lethal autonomous weapons. Indeed, as the UN secretary-general clearly said, many of us find them morally repugnant. It seems therefore that lethal autonomous weapons conflict directly with the Martens Clause.

The other three principles are also violated by lethal autonomous weapons. For instance, we don’t know how to build weapons that can adequately distinguish between combatant and civilian. The Kargu drone deployed on the Turkish–Syrian border uses facial-recognition technology to identify targets. And yet we know that, in the wild, such facial-recognition software can be incredibly inaccurate.15 It is hard, then, to imagine how the Kargu drone upholds the principle of distinction.

What’s more, we cannot yet build autonomous systems that respect the principles of proportionality and necessity. We can build autonomous systems like self-driving cars that perceive the world well enough not to cause an accident. But we cannot build systems that make subtle judgements about the expected damage a particular weapon will inflict. Or about the humanitarian trade-offs between a variety of different targets.

I am willing to concede that some of the principles of jus in bello, like that of distinction, may be achieved by AI systems at some point in the future. In a couple of decades, for example, machines may be able to distinguish adequately between combatants and civilians. Indeed, there are arguments that machines may one day be better at upholding the principle of distinction than humans. After all, machines can have more sensors, faster sensors, sensors that work on wavelengths of light that humans cannot see, even active sensors, like radar and LIDAR, which work in conditions that defeat passive sensors like our eyes and ears. It is plausible, then, that killer robots will one day perceive the world better than we humans can.

However, there are other principles, such as the Martens Clause, that it is hard to imagine machines will ever be able to uphold. How will a machine understand repugnance? How can a machine determine the public conscience? Similar concerns arise around the principles of proportionality and necessity. Could a machine ever adequately understand the humanitarian concerns that a military commander considers when some insurgents are hiding near a hospital?

In February 2020, the US Department of Defense officially announced the adoption of a series of ethical principles for the use of artificial intelligence within the military.16 The principles emerged from over a year of consultation with AI experts, industry, government, academia and the American public. They apply to both combat and non-combat situations. The ethical principles boldly promise AI that is responsible, equitable, traceable, reliable and governable.

US Department of Defense’s ethical principles for the use of AI

1.Responsible. [Department of Defense] personnel will exercise appropriate levels of judgment and care, while remaining responsible for the development, deployment, and use of AI capabilities.

2.Equitable. The Department will take deliberate steps to minimize unintended bias in AI capabilities.

3.Traceable. The Department’s AI capabilities will be developed and deployed such that relevant personnel possess an appropriate understanding of the technology, development processes, and operational methods applicable to AI capabilities, including with transparent and auditable methodologies, data sources, and design procedure and documentation.

4.Reliable. The Department’s AI capabilities will have explicit, well-defined uses, and the safety, security, and effectiveness of such capabilities will be subject to testing and assurance within those defined uses across their entire life-cycles.

5.Governable. The Department will design and engineer AI capabilities to fulfil their intended functions while possessing the ability to detect and avoid unintended consequences, and the ability to disengage or deactivate deployed systems that demonstrate unintended behaviour.

It is hard to disagree with many of these desires of the US Department of Defense. Who would want an unreliable autonomous tank that was sometimes responsible for friendly-fire casualties? Or a kamikaze drone that was biased against Black people, causing more accidental civilian deaths in Black populations than in white populations? As with other announcements of ethical principles for AI systems, two fundamental questions remain: could we achieve such laudable aims? And, if so, how would we go about it?