Chapter 10

POWER GRIDS AND POWER PLAYS

This is the largest blackout in U.S. history. If that is not a signal that we have got a problem that needs to be fixed, I don’t know what is.

—JENNIFER GRANHOLM, THEN GOVERNOR OF MICHIGAN, AMERICAN MORNING, CNN, AUGUST 18, 2003

New York is down.” It was 2003 and many people were still jumpy from the 9/11 attack that had occurred less than two years earlier. Just before 5:00 P.M. on August 14, a frantic news producer found ABC News’s Ted Koppel in a corridor and told him that the ABC television network had lost contact with the mothership, Network Control in Manhattan. “We’re running the entire national network from here in Washington now.” ABC had automatically instituted its disaster recovery plan and devolved control to D.C. “You have to go on,” the producer told Ted. “Now. Live.”

Although most Washingtonians had fled the city in the annual August evacuation to the beach, Ted Koppel was still there. For more than twenty years, Koppel had done a network television newsmagazine show, Nightline. Weekday nights the show went on the air at 11:30 P.M. East Coast time, but he taped much of it during the afternoon at ABC News’s Washington bureau. He had just completed taping that night’s show. It would never be seen.

The producer told Koppel there had been a major power blackout. New York was in the dark and so were Cleveland, Pittsburgh, and a huge swath of the Northeast. With little more information than that, Koppel sat on a stool in front of a camera and tried to explain to the rest of the country what was going on. The problem was, he didn’t know. No one did. Koppel was vamping, but, being a pro at live television, he exuded calm to an anxious audience.

Dick Clarke, then an ABC News “talking head,” had also been in the D.C. bureau when control of the national network shifted to the little building on a side street in downtown Washington. The front desk guard stopped him from leaving the building and redirected him to the studio. In a few minutes, he was sitting next to Koppel explaining to the audience how there were three isolated electric power grids (known as Interconnects) in the United States: East, West, and, well, Texas, and within them there were subregions, such as the mid-Atlantic area, known as PJM (the Pennsylvania–New Jersey–Maryland area). “It looks like the folks at PJM acted just in time to unplug us, otherwise Washington would have gone down too,” Dick suggested.

“But there is no way that this could have been terrorism, right, Dick?” Koppel asked, seeking to further assure people.

Dick paused. He knew more than he could say publicly about the fragility of the power grid in the face of a cyberattack. As the nation’s first Cyber Czar, he had pushed for meaningful cybersecurity regulation of power generation and distribution companies. The electric power lobby had pushed back hard and, largely, won. Nonetheless, President George W. Bush’s 2003 National Strategy to Secure Cyberspace that Dick had released six months before the blackout had stressed the need to increase protection of the power grid cyber controls.

Answering Koppel within the first hour of the blackout, Dick suggested that there was no way to know just then if the grid had been attacked, but it was possible, he insisted, for malicious activity to bring down the grid. Koppel downplayed the idea.

If a Tree Fell in a Forest

A U.S.–Canadian investigation completed a year later placed the blame on a tree. To be fair, it had been a really big tree, in Ohio, and it had fallen on a sagging, overloaded transmission line. Buried in the study was an observation that an internet worm running rampant that day may have slowed down the control network, possibly contributing to the cascading failures that tripped circuits providing power to 50 million people. Still, no one wanted to say that the power grid could be brought down by malicious internet activity. In retrospect, the 2003 disaster was not a cyber incident, but what Dick and others knew then was that such a catastrophe could be caused by a cyberattack.

A dozen years later, Ted Koppel was convinced too. He wrote a book called Lights Out, about the threat of a cyberattack plunging the nation into darkness not for hours, but for months. Koppel wrote that cyberattacks could cause large transformers and generators to become so damaged that they would be irreparable, and he noted that there are few spares lying around. It takes months, and electric power, to make new ones.

Even in 2015, however, some critics found Lights Out to be hyperbolic, exaggerated, or alarmist. By then, however, there should have been less doubt. In 2007, a generator was attacked and destroyed from the internet in a controlled experiment at Idaho National Laboratory. In 2015, Russian hackers plunged much of Ukraine into darkness by taking remote control of a power grid control room, making the controls reflect normal activity, and then opening breakers throughout the subgrid. A year later, they did it again, although to another part of Ukraine.

By then film, television, and thriller writers had totally accepted the idea of hackers taking down the grid. It was the premise of Bruce Willis’s Live Free or Die Hard, and the television series Madam Secretary showed an American President retaliating against Russia by plunging Moscow into darkness in winter. (By 2017, the idea of the President ever standing up to Russia seemed more fictional than any cyberattack on the grid.)

Is Self-Regulation Enough?

The electric power industry, however, continued to deny that there was much of a problem. The industry’s self-regulatory body, the North American Electric Reliability Council (NERC), was satisfied that its rules for critical infrastructure protection (known to all in the business as “nerk sip”) were adequate to protect the grid from hackers. The government’s regulatory body, the Federal Energy Regulatory Commission (FERC), tended to defer to the industry. (By 2019, however, FERC was more assertive and fined Duke Energy ten million dollars for cybersecurity lapses.)

Indeed, so powerful were the industry’s lobbyists that only part of the grid was subject to any federal regulation at all. FERC has its oversight limited to the “bulk” power system. Even then, the industry group managed to exempt 90 percent of the grid by declaring it “low impact” or noncritical.

The “last few miles” distribution systems are beyond the reach of the federal government regulators. State governments, which have shown little interest or ability in regulating the cybersecurity of power grids, have legal authority over it. Power companies have noted that the state governments, which do regulate the price for electric power, are really reluctant to authorize increases in electric rates. So really, you see, there is no way that the companies could come up with the money to pay for more extensive cybersecurity even if they wanted to, or so they say.

The FERC, however, did require power grid companies to report cyber incidents on the parts of the grid for which it has authority. In 2015 and 2016, no one did. That strained the credulity of even the FERC and the NERC, both of which began a process in 2018 to lower the threshold of incident reporting. FERC even suggested the radical notion (one commentator called it a “sea change”) that portable devices that connect to the grid, such as laptops and tablets, should meet cybersecurity standards. FERC’s newfound courage, however, may have been both too little and too late.

By the fall of 2017, the Department of Homeland Security was quietly informing power grid companies that there was good reason to believe that potential adversary nations were attempting to penetrate the controls of the U.S. power grid. For some on the receiving end of that information, it was hardly news. Some companies had been monitoring attempts to penetrate their networks for years. Others, however, seemed as if they would rather not know. They continued to repeat the mantra that they were “nerk sip” compliant, that their controls were not connected to the internet, that they could not afford to do anything more, and that it was really the government’s job to save them from foreign armies.

Blinking Red Lights

In the summer of 2018, by which time most of the nation had accepted the fact that Russia had actually meddled in the 2016 election and was hacking into anything it could in the United States, the head of the U.S. intelligence community publicly warned that the power grid had in fact already been successfully penetrated by Russia. DHS’s chief of industrial control system analysis, Jonathan Homer, specified, “They [the Russian group Dragonfly] have had access to the button, but they haven’t pushed it.” Dan Coats, the Director of National Intelligence and a former Republican Senator, described the Russian attacks on the U.S. electrical grid as being so severe that, figuratively, “the warning lights are blinking red.”

In background briefings that followed Coats’s statement, government officials explained that the Russians had “jumped the air gap,” which the power grid companies contended they had created between their internet-connected systems and the actual controls of the power networks. In point of fact, few companies had actually isolated their controls. There was almost always a path from the internet to the company’s intranet, and another path from there to the grid controls. The connections among and between the company’s internal networks were usually segmented by firewalls, but firewalls seldom stop sophisticated hackers.

By 2019, Coats and the heads of all seventeen U.S. intelligence agencies were getting more explicit. In their annual threat assessment to Congress, the agency heads wrote that Russia had the ability to disrupt the U.S. power grid and that China had the capability to disrupt the U.S. natural gas pipeline system (upon which much of the power grid relies). These were not theoretical capabilities, the agencies made clear. These were swords of Damocles hanging above America, swords that could be dropped at any time.

Russia’s hackers had allegedly gone after the companies that supply parts to or do maintenance on the grid control side of the air gap. By compromising those systems, the attackers could gain the log-in credentials of people authorized to have access to the control network. Often that access would be remote, over virtual private networks (VPNs) running on top of the internet. The Russians could then plug into the grid’s control. Then they could move into the systems that display the state of the grid on big monitors in control rooms and send instructions to the thousands of devices in the field.

If all of that sounds a little familiar to you, maybe you read speculation about how the United States had attacked the Iranian controls for the nuclear centrifuges at Natanz. The Iranians had basked confidently in the assurance that their control network was also “air gapped” from the internet. Not satisfied with the security provided by firewalls, the Iranians had sought to protect the plant from U.S. or Israeli cyberattack by having no internet connection anywhere in the complex. The United States attacked, according to some experts, by infiltrating the Stuxnet software into devices brought into the building by contractors, perhaps on laptops or printers.

Thus, the United States now faced the specter that people in American power control rooms could someday look up at “the big board,” the giant monitors on the wall, and see everything blinking green for good, while the reality was that the system was malfunctioning.

Even if you had known for years that it could happen, having the head of U.S. intelligence say that it had taken place was enough to cause shock and high levels of concern in much of official Washington. President Trump, however, did not talk publicly about the Russian presence inside the U.S. power grid controls. He could barely admit they had tried to influence his election, and that only on alternate days. Thus, there was no presidential directive, no ten-point action plan, no public threat to Moscow whatsoever.

Left on their own to devise a reaction, the Washington policy chattering classes suggested a variety of approaches to the problem of having a potential adversary possess the ability to throw much of the nation back into a nineteenth-century preelectric age, only worse, because this time we would be without manual devices. Keith Alexander suggested that it was the military’s job to defend the grid. He did not say how it would do that. Some commentators suggested that improved information sharing by the government would help the companies do a better job finding the Russians on their networks.

Taking a page from Special Counsel Robert Mueller’s playbook, others proposed that the United States should indict the individual Russian military and intelligence officers who had done the hacks. In addition, maybe we could seize their assets in the United States and ask Interpol to issue international arrest warrants for them.

The more robust-sounding responses came from those who believed in deterrence. If we were in the Russian power grid controls, then maybe we could scare the Russians into inaction, they said. Just like the U.S. President on that Madam Secretary episode, we could throw Moscow into wintry darkness if they tried to shut off the power in Washington, or mess with the Eastern Interconnect (everything east of the Mississippi, including Ontario).

All of those reactions struck us as pathetic and, were this not a serious business that could lead to death and war, even laughable. Russia doesn’t care if we sanction their intelligence officers, and none of them have assets in the United States anyway. They do not travel abroad under their real names, and they are never going to be caught because of an Interpol notice. Russia will not be deterred by the threat of the United States turning off their lights. If they are going to attack us, they will already have calculated that it will result in some risks and costs. Moreover, will you really feel better as you freeze on a dark night, eating cold tuna fish from a can and figuring out how to break into an ATM, knowing that your Russian counterpart is also in the dark? At least they have plenty of vodka.

The reason the Russians are in the controls of the U.S. power grid is because it is easy to be, and very useful to be. Whether or not they could actually bring the country to its knees without firing a shot, no one knows. And that is the point.

We have to admit that it’s a hard problem. The power grid is a crazy quilt of hundreds of disparate electric power companies of very different sizes and competencies. Each company has tens of thousands, if not hundreds of thousands, of devices connected to it, many of them sitting out in the open, unguarded.

We got here by ignoring the warnings that have been issued by government experts for almost twenty-five years that the power grid was becoming vulnerable to cyberattacks. Those warnings were ignored because such an attack had never happened before, what Dick, in the 2017 book Warnings, called the Initial Occurrence Syndrome bias.

Taking the warnings seriously would have been an inconvenience to grid owners and operators. It would also have meant spending money, which would have had the effect of raising electricity rates and/or lowering company profits.

Moreover, because self-regulation usually results in minimal regulation, really addressing the problem would have meant having serious, mandatory government regulations and compulsory compliance enforcement. Business leaders tend to resist any government regulation. They resist new regulation of any kind like white blood cells attacking a disease. “How would government know what to do? They can’t even protect their own networks.”

We have placed the national security task of creating cybersecurity for most of the power grid into the hands of fifty state-level electric-rate-setting regulators, some of whom just might be subject to the blandishments of big utility companies and their lobbyists.

Five Not-So-Easy Pieces

The more important question which you are now asking yourself is, How do we get out of this mess? Well, we begin by admitting we have a problem, a big one. There is no need here to once again paint the picture of what could happen. By now, you can imagine it. If we have a big problem, it may require a willingness to engage in bold solutions. Half measures are not recommended. We have tried that already and they have failed. So . . .

First, put someone in charge and give them real authority. We suggest that this someone be a senior official in Homeland Security. If you feel better with someone from the Energy Department, fine. If you think the military should do it, read chapter 12 on the military and realize that they are busy enough trying to defend themselves.

Real authority means federal-level mandatory and compulsory regulatory directive capability, unencumbered by prolonged legal review processes. It means authority over the cybersecurity of every aspect of the power grid: big power companies, little cooperatives, bulk power, generation, distribution, and last-mile access. It means the authority to raise rates and to direct spending. Did someone tell you security was cheap? Would you rather buy everyone an emergency generator and months of fuel?

Second, launch a major program using the best private-sector threat hunter firms to find and remove foreign implants, backdoors, and remote access to the industrial control systems (ICS) and supervisory control and data acquisition systems (SCADA) on the grid. This will not be easy. Ask the U.S. Navy how easy it was to get the Iranians out of their network (and by the way, the Russians are better).

Third, put in place that combination of state-of-the-art cybersecurity best practices that have achieved success in America’s most secure corporations. Private-sector expert panels can design the essential set of controls, but they are likely to include permanent threat hunting software and teams, continuous monitoring applications, privileged access management controls, microsegmentation, endpoint detection, remediation systems, limited remote access, and vendor/supply-chain controls. Private, third-party teams, incentivized to find vulnerabilities, must then be used on a near constant basis to monitor compliance. A pattern of noncompliance would result in severe fines, or forfeiture of the property.

Fourth, prepare better for the worst. We also need to be discussing how to maintain society once the grid goes down for a long time. There needs to be a contingency plan that helps to mitigate the most severe effects of such a worst-case scenario.

It may not be possible to stop the lights from going out from an attack that essentially flips breakers, but it should be possible to put better controls on transformers and generators to prevent a hacker from overriding their limits and causing physical destruction. In the event that fails, there should be replacement systems stockpiled, or many more generators on standby than are currently used day to day. Those spares should not, however, be used on peak days. They would be dedicated to respond to hostile attack situations to bring the grid back up in isolated subgrids.

Fifth, you want to hack into the Russian power grid’s controls and publicly threaten them with retaliation? Sure, okay, but we may already have tried that, and if so, it has not deterred them. Maybe we could say we are serious this time. And maybe deterrence will work, and maybe we can secure the existing grid with retrofits, but it would be nice to have a plan B in case deterrence fails and we cannot get all of the power companies to secure their networks.

We don’t really have a plan B today, and that contributes to crisis instability. There is probably greater uncertainty on our part about what destruction they can do to us than there is uncertainty on their part about that same question. As Dr. Strangelove might have said, “Vee need to close the uncertainty gap.” We can do that by stepping up our cyber defenses on the power grid and causing the Russians and others to be uncertain what the effect would be if they tried a major attack.

What would a plan B look like? It would be a secure, segmented, diverse-source microgrid (SSDM) program, with some microgrids completely federally funded and some built with incentives from the federal government. To build a bipartisan coalition for the program, it would be justified by both national security and climate change concerns. What we are suggesting for a plan B is:

Our SSDM proposal is at once evolutionary and revolutionary. The evolutionary aspects include the construction of large alternative-energy plants. There already are large solar and wind plants built by the private sector. Hydrogen fuel-cell plants are being built, including a 20-megawatt generator we were briefed on that would use an abandoned factory in New Britain, Connecticut.

The DoD has already built large-scale solar and geothermal plants on its bases, such as a 16-megawatt solar plant at Davis-Monthan Air Force Base outside Tucson, a 14-megawatt solar plant at Nellis Air Force Base outside Las Vegas, and a 170-megawatt geothermal plant at China Lake Naval Air Weapons Station in California.

The revolutionary parts of the SSDM proposal are threefold. First, it would be the building of a new, second national power grid on a crash basis as a major government initiative, with significant private-sector involvement. Second, the new grid would not be interconnected, but would instead consist of thousands of energy sources intended only for specific facilities. Not being interconnected, or connected in any way to the internet, it could not be taken out by a single or even a handful of cyberattacks. Third, and most important, perhaps, it would be designed with cybersecurity in mind, rather than as a grudgingly added retrofit and afterthought.

Yes, it would be expensive, but think of it as a weapons system. Without a system like SSDM, the nation will be defenseless against a nation-state actor, somebody like the Russian GRU, engaging in a cyberattack that would technologically revert us to the nineteenth century, but without all the equipment that people in the nineteenth century had to deal with life in a society without electricity.

We spent $201 billion to fund the DoD Missile Defense Agency between 1985 and 2017. In 2018 alone, Congress approved $11.5 billion for missile defense. Yet no defense expert we talked to thinks that the United States could stop a Russian missile attack on America. By saturating the defenses with multiple, simultaneous launches and scattering decoys, Russia or China could easily defeat the missile defense system. Nonetheless, we spend because we just don’t like the idea of Russia or some other nation being able to attack our territory and knock the legs out from under our society and economy. Now, remember earlier in the chapter when we quoted the Director of National Intelligence saying the Russians were hacking into the controls of our power grid?

We’ve spent enough time between us in the Executive Office of the President to know how that game is played, so we have a pay-for. If you don’t care for ours, get your own. Ours is the Air Force program to replace the Minuteman intercontinental ballistic missile (ICBM). They are planning to spend upwards of $140 billion to do that, and you just know there will be a cost overrun. We do not need to have ICBMs, land-based nuclear missiles sitting in holes on farms in Wyoming with about four hundred nuclear warheads. They are a relic of an age when we thought seriously about engaging in the kind of thermonuclear war that would kill off the population, all of it. For that purpose, we also have five times as many nuclear warheads on missiles on submarines. There are also bombers and cruise missiles.

Which kind of Russian attack is more likely: one that crashes our power grid (the power grid they have already penetrated) and causes our society to effectively cease functioning for months, or one that uses nuclear weapons and risks our nuclear retaliation and the end of all human life? Don’t struggle with the answer. It was a rhetorical question.

We are not saying abandon all efforts to secure the existing grid. We are saying that probably won’t work too well and we need a plan B, soon. A plan B that gives us another grid that the Russians could only have low confidence in damaging significantly, a second grid that would keep some of our most essential systems able to operate even if the Russians take down our Interconnects.

Some of you may be thinking that the government should not be regulating cybersecurity because it can’t even secure itself. You would be half right, as we see next.