Chapter 13
Planning for Various Disaster Scenarios
In This Chapter
Getting your business ready for natural disasters
Preparing for man-made disasters
Natural or man-made disasters throw in wild cards that can make carrying out recovery operations more difficult for your organization. In this chapter, I discuss the primary and secondary effects from a variety of disaster scenarios, and I explain how you can improve your recovery plans for the best chances of success.
Planning for Natural Disasters
Nature can throw a lot of surprises that make planning, rescue, and recovery operations difficult. Violent natural events can directly effect property and equipment, and they can also have secondary effects on communications and transportation systems, which hamper recovery efforts and may have greater impact than the event itself.
Earthquakes
Earthquakes strike with little or no warning and cause widespread damage. The violent side-to-side motion in an earthquake can cause considerable damage to buildings, equipment, and IT systems. Earthquakes often damage transportation infrastructure, particularly bridges and elevated roadways in large cities, requiring extensive repairs that can take weeks or months to complete.
Generally speaking, earthquakes occur in areas with a history of them. An earthquake rarely strikes an area that doesn’t have prior earthquake history.
Transportation considerations
When a major earthquake strikes a city, people will be spending the next several hours wherever they happen to be at the time. Transportation is usually the hardest hit infrastructure, particularly bridges, tunnels, and elevated roadways.
After human life, transportation systems are often the next most important attention-getter in an earthquake. Emergency supplies in appreciable volumes arrive by truck or rail. In severe situations, authorities and aid organizations can airlift emergency supplies, but airlifts usually bring only the absolute necessities — drinking water, food rations, and emergency medical supplies. Red Cross helicopters don’t bring in Sun servers or Cisco routers — not by a long shot!
Communication considerations
Communications networks tend to be highly congested after an earthquake, both because of damage to facilities and high usage. Many people make a quick phone call to say, “We just had an earthquake, and we’re okay.” Other people use communications facilities for longer periods of time to convey more detailed information.
The diversity in voice and data communications presents some opportunities after an earthquake. Because carriers’ networks are often located in different physical locations, one or two may suffer significant outages in an earthquake, and others may fare a little better. Text messages may fare better than voice communications over congested cellular networks.
Recovery considerations
You may want to incorporate some of these items into your DR plan if any of your business locations are at risk for earthquakes:
Emergency supplies: In the event that employees are at the workplace when an earthquake occurs and can’t travel home for a few days. Also, keep supplies that disaster response personnel need when they do arrive at the business location. Emergency supplies should include food, water, medical supplies, and blankets.
Equipment protection: Even if building codes don’t require it, protect equipment with extra bracing and other means so storage racks don’t fall down in an earthquake and equipment doesn’t fall off racks and shelves.
Emergency power: If public utilities are damaged, but your processing facility is otherwise workable, you may be able to use generator power to keep the work location running. In a severe earthquake, however, fuel trucks may have a difficult time working their way around damaged roadways to deliver fuel for the generator.
Supplemental communications: Landlines and cellular networks may be congested or damaged. For really critical needs, consider satellite phones from Iridium, Inmarsat, or Globalstar, and VSAT network connectivity. A hand-crank or solar-powered radio can also help you receive news from the outside world.
Replacement IT systems: In the event that an earthquake damages some IT systems. Include replacement user workstations because systems in employee work areas aren’t usually braced, so they may fall during the quake and be damaged.
Wildfires
Many factors contribute to the amount of warning you receive before a wildfire disaster strikes — from as little as several minutes to a few days. Transportation infrastructure may be blocked if fires approach roadways. Communications and electricity may be cut off if fires burn through areas that contain wooden power poles.
Because of the threat to human life and property, local authorities may order the evacuation of personnel if a wildfire threatens a facility. You usually have only enough time to quickly gather personal belongings before leaving. Staff don’t know how quickly they can return to the facility or whether public utilities will be cut off during their absence. You might have time to quickly gather backup tapes before leaving, but a DR plan can’t count on those tapes being rescued because a wildfire can occur after hours. In that case, personnel may not be able to reach the facility because roads may already be closed.
Transportation considerations
Although roads may be closed for several hours to a few days, they’re rarely damaged. And in most situations, personnel don’t find themselves as stranded as in an earthquake. More often, roads are closed due to poor visibility caused by heavy smoke or to provide easier access for fire fighting crews. But road closures may prevent personnel from leaving or entering the facility for a day or more.
Communication considerations
When wildfires damage communication facilities, you may have to wait at least a few days before communication is restored. So, although your facility may escape direct damage, you might be forced to go without communications for as long as several days.
Recovery considerations
Consider adding these measures to a DR plan for facilities in areas threatened by wildfires:
Emergency supplies: In the event that employees are at the workplace when a wildfire strikes and can’t travel home for a few days. You may want facemasks and/or air filters in case a lot of smoke blows toward the facility. Have disaster response personnel bring additional emergency supplies in case you exhaust the supplies stocked in the facility before their arrival.
Firefighting equipment: Employees stranded at a work facility with an approaching fire can use saws, shovels, and other gear to remove flammable materials from around the building.
Supplemental communications: A wildfire may damage landlines and cellular networks. For really critical needs, consider satellite phones and VSAT network connectivity. You can also use a hand-crank or solar-powered radio to stay in touch with the outside world.
Volcanoes
In most cases, volcanoes provide warning before they erupt — but not always. Volcanic eruptions can be violent and devastating when they do occur, so personnel at a work location should take every means and opportunity to evacuate if ordered by civil emergency authorities or law enforcement. Just get the heck out!
Volcanic eruptions are characterized by huge clouds of smoke and ash, lava flows, pyroclastic flows (fast-moving currents of hot gas, ash, and rock), and significant landslides and debris flows that can travel dozens of miles over land and through river valleys.
If any office location or workers reside within one hundred miles of an active volcano, gather information from local civil authorities regarding precautions, preparations, and evacuation procedures. I live within twenty miles of one active volcano and eighty miles from three others, so I made myself familiar with this information for my location. But volcanoes in different parts of the world have vastly different behavior, so what’s good for me may not be much help for you.
Transportation considerations
If you live or work near a volcano that may erupt with little notice, you need to be able to evacuate quickly and take next to nothing with you. If an active eruption is about to take place, staff may not be able to return to the work facility for days or weeks, and worse yet, an eruption can damage or completely destroy homes and work locations.
Recovery considerations
If a volcano is located near a primary work location, the entire business may have to start over in a different location if a serious eruption occurs. Discuss the actual risks associated with a specific volcano with civil authorities and insurance company officials.
Floods
Floods typically occur as a result of excessive rainfall in a short period of time or unusually warm weather that causes a heavy springtime snowmelt. Floods can threaten a business in several ways:
Direct building damage from flood waters: If a work location is near a stream or river, flood waters may threaten the actual structure, as well as any assets and equipment located inside.
Damage to transportation systems: Even if employees’ homes and the work facilities are on high ground, major nearby transportation systems might not be. Flood damage to nearby highways can force some workers to stay home (or at work) and disrupt the shipment of supplies, materials, and products.
Damage to communications and public utilities: Flooding may damage and cause widespread interruptions in communications and public utility infrastructures. So, although a particular business may escape the direct effects of flooding, a communications or electric power outage may still render a business location inoperative for several days.
Forced evacuations: Civil authorities or law enforcement may require that you evacuate work locations and residences to prevent loss of life during a flood.
Transportation considerations
Floods can cover or wash away roads and bridges, paralyzing transportation for many days. People get stuck wherever they were when the flood waters rose, whether at home, work, or elsewhere.
Communication considerations
In areas in which flooding occurs, communications may not be severely disrupted, although outages may still occur. Even in rivers that do flood from time to time, the characteristics of each individual flood may vary enough to make damage and disruption less than predictable.
Recovery considerations
You might incorporate some of these items into your DR plan if any of your business locations are at risk for floods:
Emergency supplies: In the event employees are at the workplace when flooding occurs and can’t travel home for several days. Also, disaster response personnel need supplies when they arrive at the business location. Emergency supplies should include food, water, medical supplies, and blankets.
Supplemental communications: Landlines and cellular networks may be congested or damaged. For really critical needs, consider satellite phones from Iridium, Inmarsat, or Globalstar, and VSAT network connectivity.
Emergency power: If floods have damaged power systems, but a processing facility is otherwise workable, you may be able to use generator power to keep the work location running. In a severe flood, however, fuel trucks may not be able to deliver fuel for the generator.
Replacement IT systems: In the event that flooding damages some IT systems. Be sure to include replacement user workstations.
Alternate work locations: You may not be able to use buildings damaged in a flood for days or weeks. The organization may need to do business elsewhere for quite a while, and you may not be able to enter the premises to retrieve important equipment or records.
Wind and ice storms
Although they’re far different from each other, wind and ice storms inflict similar damage to buildings and property. Although you usually get some warning before a wind or ice storm, you don’t have enough time to do anything but pack a few things and get to wherever you want to be stuck for a while — work, home, or someplace else.
Probably the biggest effect of wind and ice storms is widespread power outages caused by one of the following:
Wind blowing over trees, which then damage power and phone lines
Ice that causes trees to collapse onto power and phone lines
Ice that directly damages power transmission systems and telephone networks
Transportation considerations
In wind and ice storms, many trees fall over, often blocking roadways, making travel slow and difficult. In ice storms, roads are also icy. Civil authorities or law enforcement may close roads. Hence, people get stranded wherever they are.
In severe situations, delivery of supplies may be interrupted for days at a time. Workers can’t get to work (or get home from work).
Communications considerations
Communications can be hard hit in wind and ice storms. Where the storm doesn’t directly damage phone lines themselves, widespread power outages can cause cell sites and remote nodes to exhaust their emergency battery supplies. Locations that have emergency generators can work for a while after emergency battery supplies run out, but even these generators fail when they run out of fuel. In severe storms, communications can be out for several days — even a week or more.
Recovery considerations
DR plans for businesses located in areas subject to wind and ice storms might consider the following:
Emergency supplies: You may need significant stocks of food, water, medical supplies, and blankets in areas hit by these storms. Recovery personnel who can make it through should bring additional supplies if they can obtain them.
Supplemental communications: Landlines and cellular networks may be congested or damaged. For really critical needs, consider satellite phones from Iridium, Inmarsat, or Globalstar, and VSAT network connectivity.
Emergency power: Generator power may keep the work location running for a few days. But in a severe wind or ice storm, the power may be out for so long that generators run out of fuel, and fuel trucks may not be able to deliver fuel for the generator. In severe situations, damaged transportation systems may hamper emergency services and delivery of all kinds of emergency supplies.
Hurricanes
Hurricanes are the perfect storm: They include rain, wind, and flooding, each in potentially cataclysmic proportions. Big Category 4 and Category 5 storms cause widespread, severe damage that can take years to recover from. Hurricanes Camille (1969), Andrew (1992), and Katrina (2005) are prime examples of the devastating power of hurricanes.
Transportation considerations
Historically, evacuations ahead of hurricanes are slow and take days because everyone takes to the roadways. All staff members should evacuate when ordered to by civil authorities so they don’t have to remain behind to ride out the storm. Because of the colossal potential for damage and loss of life, businesses should emphasize the need to just leave when so ordered.
Communication considerations
Communications after a hurricane can be out for several days, even a week or more. Even if phone lines aren’t damaged, power outages cause cell sites and remote nodes to exhaust their emergency battery supplies. Cell sites and remote nodes that have emergency generators can work for a while after their batteries run out, but eventually even the generators fail when they run out of fuel.
Recovery considerations
Businesses that lie in the path of a hurricane should hope for the best but prepare for a complete loss of all business assets. Businesses in hurricane-prone areas may want to consider the following in their DR plans:
Emergency supplies: For those unfortunates left behind to ride out a hurricane, emergency supplies need to include food and water, medical supplies, and rain gear — enough for several days, at least.
Supplemental communications: Hurricanes can completely knock out landline and mobile phone systems. For really critical needs, consider satellite phones that completely bypass landline and cellular systems, and VSAT network connectivity. A hand-crank or solar-powered radio can help you receive news from the outside world.
Emergency power: You may need only generator power to keep the work location running for a few days. But in a hurricane, the power may be out for so long that generators run out of fuel. Remember that fuel trucks may have a difficult time delivering fuel for the generator.
Alternate processing facility: Locate a recovery center far away from areas threatened by hurricanes and have systems at this recovery center assume duties. The best time to invest in an alternate facility is after the hurricane season is over, when demand for such facilities should be lower.
Tornadoes
Extreme winds are associated with tornado funnel clouds during severe thunderstorms in the U.S. Midwest and other parts of the world. Tornadoes strike with little or no warning, and they can completely destroy a residence or business location.
The area affected by an individual tornado is quite small — usually just a few square miles. But portions of the U.S. and other parts of the world are struck by tornadoes at a disturbing rate. Unless you moved there yesterday, if you’re in tornado country, you probably already know it.
Transportation considerations
Because tornadoes strike and travel fairly quickly, wherever a person is when he or she receives the tornado warning is where he or she will be when it strikes. Many people get into their vehicles and attempt to out-run or out-maneuver a tornado, but trying to change locations during a tornado can be risky — and often fatal. Stay in the building you’re in when you receive the tornado warning and head for a designated safe area where occupants should gather during a tornado.
Communication considerations
Because tornadoes are so localized, any communications outages are usually repaired quickly, within a day or two at the most. Most cell tower structures and telephone company switching centers are built to withstand the heavy winds of a tornado. However, damage can occur, causing localized outages that can last hours to days.
Recovery considerations
Businesses located in tornado-prone areas should be prepared for the total loss of business facilities. But because of the highly local nature of tornadoes, you probably don’t need to locate your backup facility more than a few dozen miles away (unlike earthquake and hurricane disasters, which strike much larger regions). If your business is in an area prone to tornadoes, consider putting the following preparations in place:
Emergency supplies: Emergency supplies need to include food and water, medical supplies, and rain gear — enough for at least two days.
Supplemental communications: Tornadoes can completely knock out landline and mobile phone systems. For really critical needs, consider satellite phones that completely bypass landline and cellular systems, and VSAT network connectivity. You may also want a hand-crank or solar-powered radio to receive emergency communications.
Emergency power: Generator power should keep the work location running for a few days, even if workers may be away from the facility.
Alternate processing facility: You may want to set up a recovery center several miles away from your main business site if your business uses highly time-critical processes.
Tsunamis
Undersea earthquakes (and occasionally undersea volcanoes) cause tsunamis, great ocean waves that strike coastlines. Many parts of the world have inadequate warning systems, or they don’t have any warning systems at all, which puts millions of lives and property at risk of destruction. The 2004 tsunami in the Indian Ocean provides a vivid reminder that tsunamis are largely unpredictable and can exact a huge toll on lives and property.
Katrina, the lesson in preparation
The great Katrina hurricane of 2005 was a watershed natural disaster event in the U.S. The most pronounced effect from Katrina was widespread and persistent flooding when several levees broke during a storm surge. After more than two years, over 200,000 people were still displaced, and few of those displaced persons will probably ever return.
The scenario that took place was two-fold: First, the widespread severe flooding damaged buildings, records, and equipment. Businesses had little time to pack up assets ahead of the storm. Nearly everything they left behind at street level was destroyed. Second, public services were down for many days — transportation, electricity, and other utilities were down for many days; the airport was closed for more than two weeks; roads were closed for many days; and authorities didn’t permit people to return to some parts of New Orleans for weeks.
Businesses that relied heavily on an online presence that was served only in New Orleans had to relocate their online presence to other locations; businesses that had offsite backup tape storage (or could evacuate their backup media ahead of the storm) rebuilt their online presence with little or no advance planning.
Impaired communications was perhaps the most critical impact of Katrina. Few New Orleans businesses had satellite phones or VSAT (Internet via satellite) terminals.
Transportation considerations
With little or no advance warning, transportation systems become clogged beyond capacity before a tsunami. The tsunami itself may cause significant damage to transportation systems, especially elevated highways and railroads that are located at low elevations near coastlines. Repairs may take weeks or months.
Communication considerations
A tsunami may partially or completely destroy communications facilities at low elevations near coastlines, requiring a complete rebuild that may take weeks to months. Mobile carriers can compensate by bringing COWS (Cell sites On WheelS) into hard hit areas, permitting some mobile communications capabilities. Satellite phones are expensive, but you may be able to justify the cost.
Recovery considerations
You may want to incorporate some of these items into your DR plans if any of your business locations are at risk from a tsunami:
Emergency supplies: Disaster response personnel may need supplies when they arrive at the business location after the tsunami. Emergency supplies should include food, water, medical supplies, blankets, and hand-crank or solar-powered radios.
Emergency power: If public utilities are damaged, but your processing facility is otherwise workable, generator power may keep the work location running. You need to determine whether fuel trucks can work their way around damaged roadways to deliver fuel for the generator.
Supplemental communications: Landlines and cellular networks may be congested or damaged. For really critical needs, consider satellite phones from Iridium, Inmarsat, or Globalstar, and VSAT network connectivity.
Replacement IT systems: In the event that some IT systems (both servers and workstations) are damaged.
Landslides and avalanches
Earthquakes, erosion, ocean waves, groundwater, or heavy rain can cause landslides (the sudden movement of earth and rock). Several man-made causes include vibrations from machinery, construction, blasting, logging, overgrazing, and mining. Avalanches occur when an excessive buildup of snow leads to that snow, along with trees, rocks, and earth, rapidly cascading down steep slopes.
Damage from landslides and avalanches range from disrupted transportation and communications to direct destruction of homes and buildings, as well as loss of life. Recovery efforts to clear away material and debris can take hours, days, or longer.
You can easily avoid the risks from landslides and avalanches by not locating your business near the foot of a steep hillside. However, even if you locate your business away from these places, landslides and avalanches can still disrupt transportation and communication, which can have a devastating effect on business operations. Figure 13-1 shows the effects of a landslide in a major city.
Transportation considerations
Landslides and avalanches can cause transportation byways to be blocked for days, weeks, or longer. These blockages can make reaching affected areas time-consuming and difficult, increasing shipping rates and economic losses.
|
Figure 13-1: December 1999 debris-flow damage to the city of Caraballeda, on the north coast of Venezuela. |
|
Source: L.M. Smith, Waterways Experiment Station, U.S. Army Corps of Engineers
Communications considerations
Landslides and avalanches can damage communications facilities by destroying cable systems and buildings that contain communications equipment. Landslides and avalanches can affect both landline and mobile communications because most mobile communications backhaul (the communications from cell towers to switching centers) that occurs over copper or fiber optic cables on overhead structures or buried in the ground may be damaged. Satellite telephones continue to function, of course, but your business may not be able to justify the cost.
Recovery considerations
You may want to incorporate some of these items into your DR plans if any of your business locations are at risk from landslides or avalanches:
Emergency supplies: Emergency supplies need to include food, water, and medical supplies — enough for several days, particularly in locations with only a single route in and out. Disaster response personnel may need supplies when they arrive at the business location. Also consider including blankets and hand-crank or solar-powered radios in your emergency supplies.
Emergency power: If public utilities are damaged, but a processing facility is otherwise usable, generator power may keep the work location running. You need to determine whether fuel trucks can work their way around damaged or blocked roadways to deliver generator fuel.
Supplemental communications: Landlines and cellular networks may be congested or damaged. For really critical needs, consider satellite phones from Iridium, Inmarsat, or Globalstar, and VSAT network connectivity.
Pandemic
A global pandemic hasn’t occurred for so long that, until the early 2000s, many people didn’t even know the meaning of the term. The Asian SARS (Severe Acute Respiratory Syndrome) outbreak in 2002–2003 and the avian influenza threat are forcing organizations to begin contingency planning for a possible pandemic.
Pandemics in history
A pandemic is the outbreak of an infectious disease that spreads over a large region, even worldwide. Well-known pandemics in history include
Black Death in the 1300s: This outbreak of the bubonic plague killed over 25 million Europeans in six years, a quarter of the entire population. Up to half of the population died in the worst-hit urban areas.
Spanish flu, 1918–1919: This epidemic of influenza killed between 25 and 50 million people worldwide.
Asian flu, 1957–1958: Responsible for about 70,000 deaths in the U.S.
Hong Kong flu, 1968–1969: Influenza A caused about 34,000 deaths in the U.S.
Health experts at the World Health Organization (WHO) and the U.S. Centers for Disease Control (CDC) report that pandemics occur regularly throughout history and that we’re due for another. Possible candidates include the H5N1 avian influenza that has already claimed hundreds of lives, SARS (which may re-emerge at any time), tuberculosis, Ebola, and others.
A different kind of disaster
A pandemic is a widespread disaster that seems to occur in slow motion, compared to other disasters I discuss in this chapter. If and when the next pandemic hits, it’ll probably play out over several months to a few years. A pandemic is a quiet disaster: Buildings aren’t destroyed nor communications systems compromised, but the effect is nonetheless profound. (See Figure 13-2.) me of the characteristics of a pandemic include
High rates of absenteeism: More than 25 percent of workers may be absent for weeks at a time during a pandemic. Workers are either sick themselves, caring for sick family members, caring for children whose schools are closed, or fearful to venture out of their homes. This reduction in available workforce creates a general slowdown in the output of businesses across entire regions.
Unlike other types of disasters, you can’t rely on contract and other outside help to supplement absent employees. All businesses in a given region have a simultaneous need, and even contracting firms experience high absenteeism in a pandemic situation.
Supplier shortages: The availability of virtually every type of goods diminishes because the organizations producing those goods are also experiencing high absenteeism. You may have to deal with shortages of fuel, food, and other essentials for daily subsistence.
Degraded services: Levels of service may dramatically decline when you have fewer personnel available. Repairs of public utilities take longer; fewer doctors, bus drivers, and policemen are available to assist you.
Reduced demand for non-essential goods and services: Demand for items that aren’t essential for subsistence decline, causing business downturn in many sectors.
Degraded medical care: Hospitals may have to turn away most cases of illness, caring only for those who are the most ill but still have a chance of survival. Gymnasiums, conference centers, and other gathering places may be turned into gigantic hospital wards.
Forced quarantines of entire communities: Hard-hit areas experience quarantines, and government health authorities don’t permit workers to report to their workplaces for fear of spreading disease.
Closures of schools and public assemblies: Schools and other public assemblies provide opportunities for disease to spread; hence, schools may be closed for days to weeks at a time.
No outside help available: In a pandemic, large geographic regions have needs that outsiders can’t meet because those outsiders are also affected by a pandemic. Communities have to survive with the resources available locally. The International Red Cross may have to spread itself extremely thin, like a single pat of butter for a thousand loaves of bread.
Transportation considerations
Aside from forced quarantines and fuel shortages, transportation isn’t a major issue in a pandemic. Indeed, with many people huddled in their homes for fear of catching disease, roadways may actually be less congested.
Communication considerations
Organizations in a pandemic probably have healthy workers who are willing to work but can’t report to the workplace because of quarantines, lack of transportation, or sick family members at home. Businesses need to invest in remote access and remote telecommunications capabilities long before a pandemic strikes. During a pandemic wave, organizations probably experience high rates of remote workers accessing systems from their homes, not unlike when severe storms and other phenomena keep workers away.
|
Figure 13-2: An emergency military hospital during the influenza pandemic in 1918. |
|
Source: National Museum of Health and Medicine, Armed Forces Institute of Pathology, Washington, D.C.
Preparation and recovery considerations
In a pandemic, no natural event inflicts damage on buildings, records, or systems. Instead, those assets suffer neglect because you don’t have enough personnel available to care for them adequately. Businesses should develop contingency plans that include the following:
Lights-out (unmanned) processing centers: Probably a long-term proposition that requires considerable investment, organizations need to figure out how to keep their systems running with far fewer staff members than normal.
Increased capacity for remote data and voice access: With many workers able to work, but unable or unwilling to report to the business premises, organizations need to invest in additional remote data and voice capabilities. To prepare additional remote access capabilities, you need a lot of time and resources.
Reduced output: Businesses need to anticipate reductions in output and also reductions in demand. But some businesses actually experience an increase in demand, depending on the goods or services those businesses produce.
Cross-training: With significant staff shortages, you need to cross-train workers so available workers can carry out duties normally performed by workers who are absent.
Multi-sourcing critical suppliers: Identify which of your business’s suppliers are critical (without which business operations cease) and consider acquiring additional suppliers to improve the chances that you can get at least some reduced level of supplies during a pandemic outbreak.
Educate workers: Some employees stay away from work simply out of fear. Educating workers on the actual risks can give them an opportunity to take appropriate precautions based on known facts.
Planning for Man-Made Disasters
Man-made disasters fall into two distinct categories: those that are deliberate, and those that are the result of an error or oversight. Each type has its special challenges and issues that planners and emergency response personnel need to keep in mind so they can avoid further damage and casualties.
Utility failures
Practically all businesses are ravenous for electricity. IT systems are especially sensitive — they don’t have a tolerance for even relatively minor spikes, surges, and brownouts.
Some geopolitical locations have higher quality power generation and distribution systems than others. Quality and wealth aren’t necessarily in direct proportion. In my career, I’ve seen consistently poor delivery of electric power in affluent areas. But to be fair, some factors are out of the control of the power system operators, such as weather and geologic conditions.
Organizations that experience more than an acceptable level of power outages (for whatever reason) need to consider the following options to assure a continuous delivery of clean power to critical systems:
Uninterruptible Power Supply (UPS): Usually two systems in one. A UPS has circuitry that cleans incoming power of spikes, surges, and other noise so IT equipment receives the cleanest possible power. A UPS system also has banks of batteries that can become the primary power source for a short time — usually a fraction of an hour.
Electric generator: For power outages that last more than several minutes, you need an electric generator, in addition to a UPS, to assure power availability for as long as several days.
Fuel storage: For organizations that may experience power outages that last more than a few days (and if you can’t easily get fresh supplies of fuel), consider building a fuel storage facility. With such a fuel storage facility, you can have continuous electric power, even during prolonged outages.
I cover emergency power supplies more fully in Chapter 6 and Chapter 12.
Civil disturbances
Political and economic events can precipitate civil disturbances, including protests, work stoppages, strikes, vandalism, looting, and general mayhem. These disturbances can result in property damage, disruption to transportation, and temporary cessation of business operations. Law enforcement or military may block transportation routes or enforce evacuations and curfews. These events can make you want to be anywhere else.
Transportation considerations
During and after such periods of civil disruption, you may find transportation to and from affected areas limited by damage, barricades, closures, curfews, and fear. Workers at business locations may be stranded there for hours or days until things calm down.
Communication considerations
In more serious events, communications may be hampered or cut off, so disaster response personnel may have difficulty communicating with each other. As a result, you may have trouble getting an accurate assessment of an event’s effects on business operations. You may find having a variety of landline and mobile carriers helpful; if one isn’t working, perhaps another is. Workers may be able to use satellite phones if they can get to an out-of-doors location with safety.
Recovery considerations
It’s difficult to say what kinds of effects a large civil disturbance might have on business operations. Here are some measures you can take to mitigate the effects of civil disturbances:
Emergency supplies: Emergency supplies need to include food, water, and medical supplies — enough for several days. Disaster response personnel may need supplies when they arrive at the business location. Emergency supplies should also include blankets and hand-crank or solar-powered radios.
Emergency power: If public utilities are damaged, but a processing facility is otherwise usable, generator power may keep the work location running. You need to figure out whether fuel trucks can (and will) work their way around damaged or blocked roadways to deliver generator fuel.
Terrorism and war
The extent and ferocity of terrorism and war are largely unpredictable and chaotic. All you want to do is get out of the way!
Anything can happen in a war, and you can’t really know what kind of a contingency plan you’ll need. No wonder insurance companies don’t cover war in their policies.
The September 11, 2001 attacks on New York City and Washington, D.C. (see Figure 13-3) were unprecedented for many reasons. In the business continuity and disaster recovery professions, the attacks brought to light two scenarios that many BC and DR planners hadn’t considered:
The complete collapse of an otherwise structurally sound building
The loss of a large proportion of the workers in an organization
|
Figure 13-3: The New York City World Trade Center during the 9/11/2001 attack. |
|
Source: U.S. National Park Service
9/11 forced BC and DR planners back to the drawing board, updating their contingency and emergency response plans with these new scenarios. Really, it started a multi-year effort in which organizations took the time to rethink the adequacy of their emergency plans. Personally, I think that this extra planning brought about two results:
Off-site media storage centers got a boost.
Businesses thought more seriously about whether they wanted to locate their processing centers in what is essentially a landmark. Many businesses instead opted for processing centers (and alternate processing centers) in low-profile, unmarked buildings.
Security incidents
Break-ins, hacking incidents, Denial of Service attacks, and large malware outbreaks are significant events and have the potential to shut down the IT systems that support critical business processes.
A security incident can reach disaster levels in a number of ways:
Data corruption: If the incident causes data corruption, the organization may be forced to take systems offline until you can recover or rebuild the data. In large databases, this process can take several days, even on the fastest available computers.
Denial of Service (DoS): A concentrated attack, especially when it originates from large numbers of systems, can render a server or an entire network of servers unreachable to customers and partners. Such attacks can last for hours, days, or even weeks.
Forensics: Your organization (or law enforcement) may need to carry out forensic operations on affected systems to gather evidence for a possible prosecution. Trained personnel usually conduct forensics on quiescent systems (systems in which activity is halted) to provide stability, ensuring the best possible evidence gathering.
To reduce the effects of a security incident, consider these measures:
Alternate servers and storage systems: Place alternate systems and storage systems into service while trained personnel perform forensics on affected systems. Before you place affected systems into service, however, the organization needs to be certain that they won’t be compromised, too.
Distributed Denial of Service (DDoS) defense: Some Internet service providers offer a DDoS defense service that may help to restore connectivity to customers and partners. These solutions can cost you, but so can being offline.
9/11 and Cantor Fitzgerald
Cantor Fitzgerald was a major U.S. government treasury trading firm in 2001, accounting for one quarter of all Treasury bond transactions on the open market. This multi-trillion dollar market is of primary importance for investment trading in the world economy.
Cantor Fitzgerald was located on the 101st through 105th floors of One World Trade Center, 8 to 12 floors above the point at which the plane hit the tower. Because they were located above the impact zone, virtually none of the employees in the building at the time of the attack survived. Cantor lost 658 employees, or about two-thirds of its workforce.
In 2001, Cantor was building an alternate processing facility, but that facility wasn’t yet complete. The significant loss of life made recovery from this event especially difficult. Despite seemingly insurmountable odds, Cantor was back in operation out of its incomplete alternate processing facility within one week.
Business continuity and disaster recovery planners no doubt never imagined a scenario in which such a large portion of a company’s workforce is killed in a single event. Prior to 9/11, no single company of any appreciable size had ever experienced such an event.
Alternate network locations: If an organization is experiencing a Denial of Service or Distributed Denial of Service, you may be able to avoid disruption by relocating affected servers or networks to another logical place on the Internet. Whether this move works has a lot to do with the determination of the attackers and their ability to change their intended target.
Backup data: Replication and mirroring are great technologies that can compensate for hardware failures. However, in the case of a deliberate attack, replication or mirroring may propagate the corruption to other storage systems. If this widespread corruption occurs, you may need to recover data from a recent backup — but only if analysts can determine that the attack hasn’t affected a recent data backup.
Heterogeneous (similar) systems: If attackers attack through known vulnerabilities in software, using heterogeneous systems may prevent an attack from reaching some systems. For instance, if an application uses both a Sun system with Oracle databases and a Windows system with SQL Server databases, an attack through the Windows system can’t succeed on the Sun server. I discuss monocultures in more detail in Chapter 12.
Forensics training and tools: This measure is an up-front investment of dollars and training. With better forensics preparedness, your business can respond more quickly when you need forensic activities, as well as develop alternate strategies that can minimize impact on business operations.