2

FEARS WORTH HAVING

Many things can go wrong with narrow AI systems that can impact our safety. If we are not careful, we could end up with self-driving cars running over babies in strollers; out-of-control, bomb-carrying drones; missed cancer diagnoses; and nuclear plant meltdowns. However, these dangers can be prevented with the common sense those systems lack: with properly informed regulation and by ensuring that AI tasked with dangerous operations is fully tested before it is put into use.

AUTONOMOUS WEAPONS

The idea of applying even narrow AI to government-operated military weapons is a frightening thought for most people. Narrow AI–enabled weapons in the hands of terrorists is perhaps even scarier. The most terrifying scenario would be if AGI-based military systems were possible. AGI systems would bring in the potential for Terminator-like scenarios and other terrifying possibilities. Fortunately, AGI is not happening.

Unmanned aerial vehicles (UAVs) without AI have been used in warfare since the US began to deploy them after the 9/11 attacks. These UAVs include drones, which are controlled remotely by operators at consoles, similar to a video game. The weapons range in size from hobbyist quadcopters with an attached bomb to small aircraft with multiple missiles. The operator views the video produced by a camera on the drone, and when they see the target (which could range in size from large military installations to one individual terrorist), the operator initiates the attack. The actual attack occurs either by a small drone with an attached explosive warhead flying directly into the target or by launching a laser-guided missile from a larger drone. Between 2015 and 2019, remote-controlled or autonomous drones were used by the US, the UK, Israel, Pakistan, Saudi Arabia, the UAE, Egypt, Nigeria, and Turkey to kill people—with Turkey making exceptionally extensive use of them.1 You can find a video of a test of a Turkish quadcopter drone with an explosive warhead on YouTube.²

One particularly scary aspect of remote-controlled drones is that the technology is relatively simple and accessible to terrorist groups. In September 2019, ISIS attacked Iraqi troops with seventy drones that had parachute-mounted grenades.³

Some UAVs only require a human to trigger the launch process, after which they are fully autonomous and use heat-seeking capabilities, radar, laser, or geographic coordinates to zero in on the target.⁴ For example, the US military has relied on several systems that use radar to automatically target incoming missiles since 1977.⁵ Frightening as they are, none of these are AI-based technologies.

The use of AI computer vision technology (e.g., image classification and object recognition) significantly increases UAV autonomy. With computer vision technology, the military can decide to launch an attack that relies on computer vision to automatically analyze the camera video and identify the target. Military personnel can configure the UAV software so that, once the vision system identifies the target, the attack commences automatically. For example, the Turkish drones in that YouTube video are being outfitted with facial recognition technology that will automatically find and fire on specific human targets.6 We do not need to worry about autonomous weapons like the Terminator; however, by adding computer vision technology to UAVs, we add a means for UAVs to find their own targets without a human in the loop.

Autonomous weapons that do not require a human operator to pull the trigger have ethical uses, such as the ability to defend against an enemy that launches more missiles than there are humans available to launch antimissile weapons. However, autonomous weapons have the potential to produce outcomes such as mass destruction. They have been labeled “the third revolution in warfare, after gunpowder and nuclear arms.”⁷ An organization named the Campaign to Stop Killer Robots released the frightening Slaughterbots video of an onslaught of out-of-control drones causing mass destruction.⁸ In 2019, a former US Army ranger, Paul Scharre, published a book detailing several concerns about autonomous weapons, including the risks of weapons going out of control or getting hacked.⁹ The US is actively developing an autonomous drone program named Skyborg.¹⁰ Russia and China are also developing autonomous drones,¹¹ and China is reportedly selling them to other countries.¹²

The Campaign to Stop Killer Robots has obtained pledges from 4,500 AI researchers and tens of thousands of other individuals. They are asking tech leaders not to participate in the development of autonomous weapons. Google employees have vehemently asked the company not to participate in the development of AI-based weaponry. In 2017, one hundred AI leaders signed an open letter calling on the United Nations to find a way “to prevent an arms race in these weapons, to protect civilians from their misuse, and to avoid destabilizing effects of these technologies.”

Computer vision technology is also reducing the time spent by military analysts to sift through satellite images and drone videos. For example, the military has put a fair amount of effort into training machine learning systems to recognize missile launchers in satellite images.13

Similarly, military drones capture hundreds of thousands of hours of video footage each year.¹⁴ The military is training AI systems to detect missile launchers and other objects of military interest in these drone-captured videos.¹⁵ These uses of technology are significant time-savers for military analysts. However, a human-in-the-loop is still required to trigger the deployment of the weapons against the discovered targets.

Aside from autonomous weapons, automated systems exist that help military commanders make decisions as to whether to attack or hold fire. For example, the US Army is researching the feasibility of an AI-based battlefield advisor that can analyze data from a variety of sources, including drones, satellites, and cameras mounted on the goggles of soldiers. It can then synthesize the data and make battlefield recommendations.¹⁶ These are likely to be incremental improvements to the human-in-the-loop systems that exist today rather than systems that can autonomously press the big red button.

There are also several initiatives underway to develop a baseline of ethical principles for the use of AI in military applications, including principles established by the US Department of Defense.¹⁷ Even Russia has called for the regulation of this technology.18 The global community banded together to try to eliminate the use of biological and nuclear weapons, and many argue that we should take the same path for AI-based autonomous weapons.

CYBERSECURITY

If AGI were ever achieved and embedded in a virus, we would have much bigger cybersecurity problems than we have today. A thinking virus could navigate networks and make real-time nefarious decisions. For example, it could conceivably turn its virus characteristics on and off when necessary to evade antivirus software. Fortunately, AGI exists only in fiction.

The most obvious threat to safety is the potential for cyberattacks on autonomous vehicles or autonomous weapons. The hack itself would not be AI-based; it would be a conventional (non-AI) hack of an AI system and like a hack of a tire pressure monitoring system (TPMS) on a non-self-driving car.¹⁹ However, most newer cars can control steering and braking electronically. If a hacker accesses a vehicle through a sensor like a TPMS, we could imagine that the hacker could gain access to the steering and braking systems with potentially deadly results if the hack causes a crash.²⁰

On the positive side, AI can actually help defend against cyberattacks. The US military has developed an AI-based system that can inform an autonomous vehicle that someone hacked it.²¹ Companies like Blue Hexagon, ExtraHop, and Synack use AI to detect vulnerabilities and cyberattacks.

We also need to be alert to the possibility of adversarial attacks on other types of deep learning systems, including attacks on image and facial recognition systems, as well as on speech recognition systems.²²

Baidu researchers published a set of tools that can be used by other researchers to fool virtually any deep learning system.23 Their goal, of course, was not to encourage hackers, but rather to help researchers create defense mechanisms in their deep learning systems.

Hackers can create other types of cybersecurity threats with narrow AI technology. Some of these threats are small improvements on existing hacking techniques. Audio deepfake capabilities enable a new type of spear phishing. Spear phishing attacks mostly involve hackers sending well-researched emails to targeted individuals. Using audio deepfakes, hackers can send those same targeted individuals voice messages that sound like someone they know. If fake news AI technology ever becomes viable, hackers could use it to automate spear phishing.

The counter to spear phishing remains education. We have learned to ignore emails from Nigerian princes, and we will, unfortunately, need to stop assuming that recognizing someone’s voice is enough evidence to act on what they tell us.

AUTONOMOUS APPS

Although we do not have to worry about killer robots that can think and reason, even a narrow AI system can wreak havoc if it does its one job badly. We can imagine a hypothetical AI system failing to adjust controls in a nuclear plant to avoid a meltdown or an AI-powered medical diagnosis system failing to detect cancer. IBM’s Watson reportedly took medical data and recommended unsafe and incorrect cancer treatments.²⁴

Bad conventional software can also wreak havoc. A missing hyphen in the software ruined the 1962 Mariner space launch.²⁵ Faulty software was also the cause of the 1979 Three Mile Island nuclear disaster,²⁶ the 2003 New York City blackout,27 2010’s high-speed trading outage on Wall Street,²⁸ the 2012 loss of 440 million dollars in forty-five minutes by a financial firm,²⁹ and the 2018 and 2019 Boeing 737 Max crashes.³⁰

The point here is that any type of software can cause public safety issues if it is not thoroughly tested. AI systems are software programs that someone creates to perform a specific task. Just like other software, these systems require testing, and the degree of testing needs to be modified based on the risk profile. A software program that plays checkers does not require anywhere near as much testing as a software program that controls portions of nuclear power plants.

AI applications can be harder to test than conventionally coded applications, and it may be impossible to rigorously test some AI applications. Conventional software programs can usually be tested against a specification that exactly determines which results are correct. However, machine learning applications will perform well only on inputs that are like those in its training data.³¹ For other inputs, the application might miss a case of cancer or cause a meltdown in a nuclear plant. It can be difficult or impossible to specify which types of inputs the application should handle correctly. A software application needs to be tested to certify that it operates correctly. However, if the correct operation cannot be clearly defined, it cannot be rigorously tested or certified.

Not all conventional software is testable either.³² Using untestable software in a checkers program has little downside. Using untestable software to run a nuclear plant is another story. If an AI application is untestable, it should be clearly labeled as such and should not be relied on for critical decisions.

Philosopher Daniel Dennett, of Tufts University, argued that people should never pass the buck to AI systems.³³ Manufacturers should disclose the limitations of AI systems the same way they disclose the side effects of medications. For example, vendors of medical diagnostic systems should be required to specify the expected error rates.34 Similarly, manufacturers need to inform potential purchasers when it is not possible to rigorously test an AI application. People who use AI systems should understand the intended use of the system and be aware of its limitations—just as with conventional software.

Most importantly, if an AI system cannot be rigorously tested, then it should not be used for critical decision-making, such as running a nuclear power plant. The US Food and Drug Administration (FDA) has proposed a regulatory framework to ensure the safe use of AI in medical devices.³⁵ Regulators should similarly require manufacturers of AI systems to provide proof of rigorous test plans for any machine learning components used in them. Regulators should know whether a machine learning system can be relied on to recognize every potential danger and to deal with them.

AUTONOMOUS VEHICLES

The National Highway Traffic Safety Administration (NHTSA) has defined six levels of self-driving vehicles. Levels 0 and 1 are conventional vehicles and assistance (such as lane detection), respectively. In Levels 3–5, the vehicle is controlled by software. Level 2 is the state of the art at the moment; its capabilities include adaptive cruise control, lane centering, and automatic emergency braking. Automatic cruise control detects cars in front and slows down and speeds up to maintain a driver-specified distance. Lane centering keeps the vehicle in a marked lane. On a highway, cruise control and lane centering alone are enough to keep a car in a lane at a reasonable distance behind the vehicle in front.

Figure 2.1 NHTSA’s six levels of automation.

I own a Tesla, and, while I love the car, it is not safe to read a book while driving it, even on the highway. For example, when I go around a sharp curve, I see the curve in advance and slow down to make sure I do not skid or tip over. My Tesla does not slow down when entering a big curve, so I must manually take over control.

On the flip side, my Tesla occasionally slams on the brakes so hard that it is a good thing I am wearing my seat belt or I might hit the windshield. Usually, this happens on rural roads, but it also happened a couple of times on highways.36 Fortunately, I did not have a tailgater in either of the highway incidents. Each time my Tesla does this, I try to figure out what it thinks it “saw.” Sometimes I look all around the edges of the road. Sometimes I look at the screen. Each time, this reminds me of how object recognition systems can see things that are not there.

The Tesla has many other issues when driving on city or rural streets. It has no idea how to navigate a lane closure with a person directing traffic. Sometimes, if a lane splits, left to its own decision-making, the Tesla heads into the wrong lane, which ends abruptly at a telephone pole, and I have to perform a last-minute takeover.

Although Tesla is rated the top autopilot system by most reviewers, such as Autopilot Review, it is not ready for Level 3 use, even on the highway. Tesla’s website confirms that “autopilot is a hands-on driver assistance system that is intended to be used only with a fully attentive driver. It does not turn a Tesla into a self-driving car, nor does it make a car autonomous.”37

The first death caused by a self-driving car occurred on May 7, 2016, in Williston, Florida. A white eighteen-wheeler crossed the highway,³⁸ and, because of a white background behind the truck, the AI-driven Tesla failed to recognize that an object was crossing the road and ran right into it. Since the Tesla was only a Level 2 vehicle, it was the driver’s responsibility to pay attention to the road and take over when the Tesla autopilot missed seeing the truck. Although this accident might be attributed to AI, it was actually caused by human negligence.

A Level 2 self-driving vehicle is not fully autonomous. Although the car’s computer system does most of the work steering and plotting the route, a driver is required to be seated behind the wheel and to pay attention to the road, ready to take over should something go wrong. If the self-driving car is operating at a Level 2 capability, it is difficult for the manufacturer to be held responsible because the driver is responsible for the safe operation of the vehicle.

Uber, which was testing self-driving taxis in Arizona, had a test car strike and kill a pedestrian in March 2018 while in autonomous mode. The safety operator, who was supposed to be paying attention to the road, was watching Hulu.³⁹ Uber self-driving vehicles had thirty-seven crashes before that fatality,⁴⁰ and the company had to sweat it out for a year before they learned that they would not face criminal charges.⁴¹ However, because this was also a Level 2 vehicle, the driver was required to pay attention and prosecutors will likely file vehicular manslaughter charges against the driver. The family of the victim is also suing the state of Arizona for $10 million for policies that welcomed self-driving cars into the state.⁴²

Perhaps the biggest challenge will be setting safety standards for autonomous vehicles. Is it enough that driverless car technology progresses to the point where the vehicles cause fewer accidents and fatalities than human drivers? Or do we need to get to the point where they cause no accidents or deaths at all?

Interestingly, the answers to these questions will probably be different around the world. Kai-Fu Lee, the former head of Google China, makes a strong case that China will be more tolerant of accidents and fatalities than the US and that autonomous vehicles will take to the road sooner there than in the US.43

Many legislators around the world are pushing hard for regulations that would encourage the rollout of autonomous vehicles. Two US congressmen, Illinois Democrat Bobby Rush and Indiana Republican Larry Bucshon, penned an opinion piece in early 2020 titled “When It Comes to Autonomous Vehicles, the US Cannot Afford to Be Left in the Dust.” Some countries and municipalities will be more conservative than others. It will be interesting to see which areas are the first to allow drivers to read a book while riding in Level 3 systems.

In early 2020, the US Congress was actively considering several bills that would remove regulatory barriers for autonomous vehicles.⁴⁴ This prospect terrifies safety advocates. The US National Transportation Safety Board recommended stronger regulation of Level 3 and higher systems in its February 2020 review of a fatal Tesla crash.⁴⁵ The Insurance Institute for Highway Safety wants to see a higher degree of driver monitoring that ensures drivers are paying attention to the road on Level 2 vehicles. Their concern is that Level 2 capabilities will fool drivers into thinking they do not have to pay attention.⁴⁶ And the safety group Advocates for Highway and Auto Safety put out a March 2020 press release pointing out that there have been many self-driving car crashes. Six of those crashes resulted in fatalities, including a pedestrian walking a bicycle.47

In the US, Florida has the most liberal laws of all the states. Florida legislators passed statute 316.85 in 2016,⁴⁸ which specifically allows the operation of autonomous vehicles. It explicitly states that a driver does not need to pay attention to the road in a Level 3 vehicle (e.g., the driver can watch movies) and explicitly permits autonomous vehicle operation without a driver even present in the vehicle. Also, it requires no vehicle safety tests. Whenever a car, truck, bus, or taxi company decides they are ready, they are free to test and sell driverless vehicles. I own a home in Florida, and I am terrified at the prospect of driving next to autonomous vehicles without commonsense reasoning capabilities.

Some other states permit autonomous vehicles with varying restrictions, and many other states have legislation that encourages the testing of autonomous vehicles. For example, Arizona’s governor signed an executive order in 2015 empowering the state government to “undertake any necessary steps to support the testing and operation of self-driving cars.”⁴⁹ However, the NHTSA stepped in when a French company, TransDev, attempted to launch a driverless school bus to ferry children to school in that state.

California has stricter regulations than most states concerning driverless vehicle testing. It requires service-based autonomous vehicle companies to file a report each time there is a disengagement (i.e., when a human safety operator takes over driving control).

Big-four accounting firm KPMG puts out an annual report analyzing the twenty-five countries doing the most from both a funding and regulatory perspective to advance autonomous vehicles.⁵⁰ According to KPMG, the leading countries are the Netherlands, Singapore, Norway, and the US, in that order.

To better ensure public safety, regulators should require all manufacturers and service operators who are testing Level 3 and higher capabilities to report all disengagements. For service-based autonomous vehicles, they should review every occurrence in which a human safety operator decided to take over control of a shuttle, taxi, truck, or bus during a test drive.

Regulators should also require manufacturers of consumer vehicles with Level 2 capabilities to record and report patterns of disengagements. Currently, Tesla monitors these disengagements but does not turn over the results to regulators. Most cars with Level 2 capabilities do not even send disengagement data back to the manufacturer. The safest approach would be for regulators to require that all manufacturers adopt the Tesla model, put an always-on cellular connection in every car, and use it to monitor Level 2 disengagements.

After regulators have reviewed large numbers of disengagement reports, they should set safety standards based on what they have learned. The safety standards should include lengthy testing periods in which there are no disengagements that might have turned into accidents or traffic jams.

Unfortunately, at the time of this writing, most regulatory bodies do not even require autonomous vehicle manufacturers to file test disengagement information. California requires the filing of some disengagement information, but only for service-based autonomous vehicles.

Another option is for legislators to require manufacturers to self-certify and submit the results of a standardized test plan. A group of car manufacturers have proposed the outline of such a plan.51 Testing under controlled conditions is not as good as real-world experience, however. Although testing under controlled conditions will have a paucity of edge cases, it may be better than nothing. And I am worried that nothing may be the choice of legislators and regulators.

There is significant pressure on lawmakers and regulatory bodies to remove the barriers to autonomous vehicles. In 2016, the NHTSA and the US Department of Transportation published a position paper on autonomous vehicles titled Automated Driving Systems 2.0: A Vision for Safety. In 2020, the NHTSA titled the updated position paper Ensuring American Leadership in Automated Vehicle Technologies: Automated Vehicles 4.0. The emphasis has shifted from safety to innovation leadership. Legislators and regulators, in their rush to embrace autonomous vehicle technology, may allow unsafe autonomous vehicle operation, and the results will be disastrous. The question of when autonomous vehicles are safe enough should not be decided by the manufacturers. Lawmakers and regulators must keep the public safe. They need to step in, set standards, and make these decisions.

Safety advocates are also concerned that, in their push to smooth the way for autonomous vehicles, lawmakers will absolve vehicle makers of self-driving liability. Suppose an autonomous vehicle misclassifies a baby stroller as a piece of paper, and the result is a fatal accident. Or suppose a terrorist hacks into a group of cars and causes them to crash. We cannot allow the manufacturer to say, “Sorry, that was a bug. We will fix it soon,” and walk away from any further responsibility.

Regulators should place liability for accidents caused by vehicles operating at Level 3, 4, or 5 (autonomous modes) on the shoulders of the manufacturers. Legislators also need to determine whether the suppliers of autonomous vehicle capabilities have any liability in addition to—or instead of—the vehicle manufacturer. The possibility of significant liability will hopefully cause the manufacturers to delay the widespread rollout of self-driving vehicles until the technology is safe.

Removing liability is dangerous because autonomous vehicles have no commonsense reasoning capabilities. Without these capabilities, autonomous vehicles cannot avoid causing fatalities, property damage, and traffic jams if they are allowed to roam our streets and highways at will. If legislators absolve automakers from liability for accidents caused by Level 3 and higher operation, there will be no financial barrier to manufacturers putting unsafe vehicles on the road.

A requirement that vehicles with autonomous capabilities have black box data recorders like the ones in airplanes would also be prudent so that we will be able to diagnose the causes of the inevitable accidents quickly.

Lastly, autonomous vehicles raise ethical and practical dilemmas. If a driverless car must choose between injury to its car’s passenger and the passenger of another vehicle (or a pedestrian), what choice should it make? How should a car decide between one action that kills a mother and baby and another that kills five men?52 More importantly, are we comfortable leaving that decision in the hands of a computer with no commonsense reasoning capability?

LIABILITY, TESTING, AND COMMON SENSE

There are two primary threats to public safety from AI: autonomous systems and bugs. Autonomous weapons increase the ability of nations and terrorists to wage war. There is a threat to public safety from intentional attacks using both autonomous and conventional weapons. However, autonomous weapons increase the possibility of out-of-control destruction. International cooperation has been somewhat successful in reducing the risk from weapons of mass destruction, and the same will be necessary for autonomous weapons. Liability laws also need to be part of the solution to encourage manufacturers to design these weapons with adequate fail-safe mechanisms.

Bugs can occur in both AI software and conventional software products. AI software can be harder to test than conventional software. Manufacturers need to factor this difficulty into their production processes, and consumers need to factor this difficulty into their evaluation of the software’s fitness for use. Machine learning systems that cannot be rigorously tested should not be used to run critical systems.

Autonomous vehicles are dangerous because they cannot be imbued with common sense, and it will be difficult for them to handle all the edge cases that they will encounter. Legislators need to put the burden on manufacturers to prove that their vehicles are safe for their intended purpose. It is also critical that they do not remove the liability burden from autonomous vehicle manufacturers and operators so that at least there is a financial incentive to only deploy safe vehicles. Perhaps most importantly, regulators need to require proof of rigorous testing of pedestrian detection, stop sign detection, and other machine learning systems that can cause fatalities if they fail.