Notes

INTRODUCTION

1. André Vauchez, Sainthood in the Later Middle Ages, trans. Jean Birrell (Cambridge, UK: Cambridge University Press, 1997); and Robert Bartlett, Why Can the Dead Do Such Great Things?: Saints and Worshippers from the Martyrs to the Reformation (Princeton, NJ: Princeton University Press, 2013), pp. 3–56.

2. Eric W. Kemp, Canonization and Authority in the Western Church (Oxford, UK: Oxford University Press, 1948), p. 35.

3. Nicholas Hilling, Procedure at the Roman Curia: A Concise and Practical Handbook, second ed. (New York: John F. Wagner, 1909).

4. John Moore, A View of Society and Manners in Italy, vol. 1 (London, UK: W. Strahan and T. Cadell in the Strand, 1781), pp. 454–455.

5. Matthew Bunson, 2009 Catholic Almanac (Huntington, IN: Our Sunday Visitor Publishing, 2008).

6. Alan Riding, “Vatican ‘Saint Factory’: Is It Working Too Hard?” New York Times, April 15, 1989, p. A4.

7. Melinda Henneberger, “Ideas & Trends: The Saints Just Keep Marching In,” New York Times, March 3, 2002, p. C6.

8. George W. Bush, Decision Points (New York: Random House, 2010), p. 421.

9. Ibid., pp. 420–421; Dick Cheney, In My Time: A Personal and Political Memoir (New York: Simon and Shuster, 2011), pp. 465–472; Robert M. Gates, Duty: Memoirs of a Secretary at War (New York: Knopf Doubleday, 2014), pp. 171–177; and David Makovsy, “The Silent Strike: How Israel Bombed a Syrian Nuclear Installation and Kept it Secret,” New Yorker, September 17, 2012, pp. 34–40.

10. Interview with Stephen Hadley, June 12, 2014.

11. Interview with Gen. Michael Hayden, January 21, 2014.

12. Ibid.

13. Interview with a former senior CIA official, May 2014.

14. Interview with Gen. Michael Hayden, January 21, 2014.

15. Interview with Stephen Hadley, June 12, 2014.

16. Interview with Robert Gates, June 24, 2014.

17. Bob Woodward, “In Cheney’s Memoir, It’s Clear Iraq’s Lessons Didn’t Sink In,” Washington Post, September 11, 2011, p. A25; and Gen. Michael Hayden, “The Intel System Got It Right on Syria,” Washington Post, September 22, 2011, p. A17.

18. Bush, Decision Points, p. 421.

19. Interview with a former senior Central Intelligence Agency (CIA) official, May 2014.

20. Central Intelligence Agency, “CIA Comments on the Senate Select Committee on Intelligence Report on the Rendition, Detention, and Interrogation Program,” Memorandum from the director of the CIA to Dianne Feinstein and Saxby Chambliss, June 2013, p. 25.

21. Ibid., p. 24.

22. David Dunning, Self-Insight: Roadblocks and Detours on the Path to Knowing Thyself (New York: Psychology Press, 2005).

23. Thorstein Veblen, “The Instinct of Workmanship and the Irksomeness of Labor,” American Journal of Sociology, 4(2), 1898, p. 195.

24. Adam Bryant, “Bob Pittman of Clear Channel on the Value of Dissent,” New York Times, November 16, 2013, p. BU2.

25. Interview with Amy Edmondson, June 3, 2014.

26. Mike Spector, “Death Toll Tied to GM Faulty Ignition Hits 100,” Wall Street Journal, May 11, 2015 [www.wsj.com/articles/BT-CO-20150511–710130]; and GM Ignition Compensation Claims Resolution Facility, “Detailed Overall Program Statistics,” updated June 26, 2015, accessed June 30, 2015 [www.gmignitioncompensation.com/docs/programStatistics.pdf].

27. Michael Wayland, “Deaths Tied to GM Traced to ‘Catastrophic’ Decision: Report Finds Automaker Lacked Accountability,” MLive.com, June 6, 2014.

28. Anton Valukas, “Report to Board of Directors of General Motors Company Regarding Ignition Switch Recalls” (Jenner and Block, May 29, 2014).

29. Massimo Calabresi, “A Revival in Langley,” Time, May 20, 2011.

30. Warren Fishbein and Gregory Treverton, “Rethinking ‘Alternative Analysis’ to Address Transnational Threats,” Sherman Kent Center for Intelligence Analysis, Occasional Paper 3(2), October 2004; and CIA, “A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis,” March 2009, publicly released May 4, 2009.

31. The CIA Red Cell, which has an unprecedented remit to do alternative analyses, should not be confused with military red cells that only take an adversary’s perspective. As Marine Corps doctrine declares: “The purpose of a red cell is to assist the commander in assessing [courses of action] against a thinking enemy. Depending on the size of the organization, a red cell can range in size from an intelligence officer to a task-organized group of subject matter experts (SMEs). While a red cell’s principal duties center on course-of-action (COA) development and the COA war game, it participates in the analysis of the [center of gravities] and also supports the commander’s understanding of the problem during the initial stages of design.” See, US Marine Corps, “MCWP 5–1: Marine Corps Planning Process,” 2010, pp. 2–6.

32. McKinsey & Company, “Red Team: Discussion Document,” presentation to the Center for Medicaid and Medicare Service, undated, p. 2.

33. Hearing of the House Committee on Energy and Commerce, Subcommittee on Oversight and Investigations, “Security of HealthCare.gov,” November 19, 2013; and Sharon LaFraniere and Eric Lipton, “Officials Were Warned About Health Site Woes,” New York Times, November 18, 2013, p. A17.

34. Interview with Gregory Pirio, July 18, 2013.

CHAPTER ONE

1. Gregory Fontenot and Ellyn Ogden, “Red Teaming: The Art of Challenging Assumption,” presentation at PopTech Annual Ideas Conference, Camden, ME, October 21, 2011.

2. Interview with Lt. Gen. Paul Van Riper, May 31, 2013.

3. Interview with Ben Gilad, December 20, 2013.

4. Interview with Mark Chussil, April 9, 2014.

5. Interview with Gen. David Petraeus, February 19, 2014.

6. Interview with Jami Miscik, May 21, 2012.

7. Interview with Gen. David Petraeus, February 19, 2014.

8. Interview with Lt. Gen. H. R. McMaster, December 4, 2014.

9. Interview with an Army colonel, June 13, 2013.

10. Interview with Ken Sawka, May 9, 2014.

11. Interview with Steve Elson, June 12, 2013.

12. Interview with Wayne McElrath, August 23, 2013.

13. Interview with Jayson Street, September 23, 2013.

14. Interview with Lt. Col. Brendan Mulvaney, May 1, 2014.

15. Interview with Charles Henderson, March 12, 2014.

16. Interview with Lt. Col. Daniel Geisenhof and a Marine Corps colonel, March 15, 2014.

17. Scott Eidelman, Christian Crandall, and Jennifer Pattershall, “The Existence Bias,” Journal of Personality and Social Psychology, 97(5), 2009, pp. 765–775.

18. Interview with Rodney Faraon, May 27, 2014.

19. Interview with Marissa Michel, October 7, 2013.

20. Interview with Col. James Baker, January 14, 2014.

21. Interview with members of the CIA Red Cell, March 14, 2014.

22. Interview with Chris Nickerson, June 12, 2014.

23. University of Foreign Military and Cultural Studies, Liberating Structures Handbook, p. 27. The handbook lists forty-three red-teaming tactics, techniques, and procedures.

24. Interview with Raymond Parks, June 10, 2014.

25. Interview with Lt. Col. Bill Greenberg, March 10, 2014.

26. Interview with Ellyn Ogden, July 10, 2013.

27. Interview with Col. Mark Monroe, March 10, 2014.

28. Interview with Capt. James Waters, March 31, 2014.

29. Interview with Jeff Moss, September 24, 2013.

30. Interview with James Miller, March 27, 2014.

31. Interview with Robert Gates, June 24, 2014.

32. Nuclear Regulatory Commission, “Frequently Asked Questions About Force-on-Force Security Exercises at Nuclear Power Plants,” updated March 25, 2013, accessed March 17, 2015 [www.nrc.gov/security/faq-force-on-force.html].

33. Interview with Jayson Street, September 23, 2013.

34. Interview with Catherine Pearce, June 3, 2014.

CHAPTER TWO

1. Karl Moore, “The New Chairman of the Joint Chiefs of Staff on ‘Getting to the Truth’,” Forbes, October 20, 2011.

2. Office of Management and Budget, U.S. Fiscal Year 2016 Budget of the U.S. Government, February 2, 2015, p. 134; Defense Manpower Data Center, “Department of Defense Active Duty Military Personnel by Rank/Grade,” updated May 31, 2015, accessed June 23, 2015 [www.dmdc.osd.mil/appj/dwp/dwp_reports.jsp]; and Defense Manpower Data Center, “Department of Defense Selected Reserves by Rank/Grade,” updated May 31, 2015, accessed June 23, 2015 [www.dmdc.osd.mil/appj/dwp/dwp_reports.jsp].

3. See the WGBH interview with RAND Corporation economist and Nobel Laureate Thomas Schelling for his colorful description of leading blue versus red nuclear war games during this period. WGBH, “Interview with Thomas Schelling,” March 4, 1986.

4. George Dixon, “Pentagon Wages Weird Backward Inning Game,” Cape Girardeau Southeast Missourian, dist. King Features Syndicate, May 31, 1963, p. 6.

5. Robert Davis, “Arms Control Simulation: The Search for an Acceptable Method,” Journal of Conflict Resolution, 7(3), September 1, 1963, pp. 590–603.

6. Interview with James Miller, March 27, 2014.

7. Joint Chiefs of Staff, Joint Publication 2–0: Joint Intelligence, October 22, 2014, p. 1-28.

8. Ibid.

9. Department of Defense, Department of Defense Base Structure Report FY2014 Baseline, 2015, p. 6.

10. Spiegel staff, “Inside TAO: Documents Reveal Top NSA Hacking Unit,” Der Spiegel, December 29, 2013.

11. Interview with Brendan Conlon, April 15, 2014.

12. Interview with an Army colonel, December 1, 2014.

13. Nellis Air Force Base, “414th Combat Training Squadron ‘Red Flag’,” updated July 6, 2014; and Interview with an Air Force colonel, November 24, 2014.

14. Mark Bowden, Guests of the Ayatollah: The Iran Hostage Crisis: The First Battle in America’s War with Militant Islam (New York: Grove Press, 2007), pp. 452–461.

15. David C. Martin, “New Light on the Rescue Mission,” Newsweek, June 30, 1980, p. 18.

16. Bowden, Guests of the Ayatollah: The Iran Hostage Crisis: The First Battle in America’s War with Militant Islam, pp. 137 and 229.

17. Department of Defense, Rescue Mission Report (Washington, DC: Government Printing Office, August 23, 1980), p. 22.

18. Interview with an Army major general, November 19, 2014; and Stephen J. Gerras and Leonard Wong, Changing Minds in the Army: Why It Is So Difficult and What to Do About It (Carlisle Barracks, PA: U.S. Army War College Press, 2013), p. 9.

19. Interview with Lt. Col. Daniel Geisenhof, March 15, 2014.

20. Rumsfeld passed over eight active-duty four-star Army generals to bring Schoomaker out of retirement. A ninth, Acting Army Chief of Staff Gen. John Keane, was offered the position, but declined it for family reasons. Schoomaker remains the only retired officer to become chief of staff in the Army’s 238 years. Donald Rumsfeld, Known and Unknown: A Memoir (New York: Penguin, 2011), p. 653; Interview with Gen. Peter Schoomaker, February 4, 2014; Interview with Gen. John Keane, September 27, 2006; and Paul Wolfowitz, “Remarks as Delivered by Deputy Secretary of Defense Paul Wolfowitz,” Eisenhower National Security Conference, Washington, DC, September 14, 2004.

21. Interview with Gen. Peter Schoomaker, February 4, 2014.

22. Ibid; and Hearing of the Senate Armed Services Committee, “Nominations Before the Senate Armed Services Committee,” July 29, 2003.

23. Ibid.

24. Interview with Col. Steve Rotkoff, March 3, 2014.

25. Hearing of the Senate Armed Services Committee, Subcommittee on Strategic Forces, “Hearings on Fiscal Year 2005 Joint Military Intelligence Program (JMIP) and Army Tactical Intelligence and Related Activities (TIARA),” April 7, 2004.

26. Interview with Col. Gregory Fontenot, February 14, 2014.

27. University of Foreign Military and Cultural Studies (UFMCS), Liberating Structures Handbook.

28. UFMCS, The Applied Critical Thinking Handbook 7.0, January 2015, accessed March 17, 2015 [usacac.army.mil/sites/default/files/documents/ufmcs/The_Applied_Critical_Thinking_Handbook_v7.0.pdf].

29. The vast majority is from the US armed services, but now American officers also can take courses as an elective while they attend the Command and General Staff College located a short distance from UFMCS at Fort Leavenworth.

30. UFMCS uses the Net Promoter Score popularized by Fortune 500 companies and consultants to determine customer feedback and loyalty.

31. Interview with Col. Steve Rotkoff, December 4, 2014.

32. Interview with an Army official, April 2014.

33. Interview with Col. Steve Rotkoff, May 21, 2012; and Interview with Col. Mark Monroe, March 10, 2014.

34. Interview with a senior civilian Pentagon official, March 2014.

35. E-mail correspondence with a member of the J-7 red team, March 2014.

36. Interview with Col. Steve Rotkoff, March 3, 2014.

37. US Marine Corps, 35th Commandant of the Marine Corps Commandant’s Planning Guidance, 2010, p. 12.

38. Maj. Ronald Rega, MEF and MEB Red Teams: Required Conditions and Placement Options, thesis for master of military studies, US Marine Corps, 2012–2013, p. 17–18; When Napoleon and his general staff met to develop battle plans, Napoleon brought in a corporal to shine his boots, knowing that the corporal would listen to the briefing. Once the meeting concluded, Napoleon would consult the corporal, asking if he understood the plan. If the corporal responded “yes,” Napoleon would command his staff to execute the plan, but if the corporal responded “no,” then the plans were redrawn. See, Dale Eikmeier, “Design for Napoleon’s Corporal,” Small Wars Journal, September 27, 2010.

39. Rega, MEF and MEB Red Teams: Required Conditions and Placement Options, pp. 16–20.

40. Gidget Fuentes, “Amos Forms Front-Line Groups to Study Enemy,” Marine Corps Gazette, December 21, 2010.

41. US Naval Institute Proceedings, “‘We’ve Always Done Windows’: Interview with Lt. Gen. James T. Conway,” 129(11), November 2003, pp. 32–34.

42. Interview with a retired Marine colonel, March 2014.

43. Interview with Lt. Col. Brendan Mulvaney, May 1, 2014.

44. Interview with Col. Timothy Mundy, May 2014.

45. Interview with Lt. Gen. John Toolan, June 25, 2014.

46. Interview with a Marine colonel, November 20, 2014.

47. Interview with Lt. Col. Daniel Geisenhof and a Marine colonel, March 15, 2014.

48. Interviews with II MEF red teamers and staff, February–April 2014.

49. Interview with Maj. Jose Almazan, March 11, 2014.

50. E-mail correspondence with Brig. Gen. Dan Yoao and I MEF red teamers and staff, May 2014.

51. Interview with Lt. Gen. John Toolan, June 25, 2014.

52. Interviews with prior and current commandant red team members, and Marine officers, 2013 and 2014; and US Marine Corps, US Marine Corps 36th Commandant’s Planning Guidance, January 23, 2015. Eventually, Amos’s own commandant’s red team was placed within the Strategic Initiatives Group (SIG), known as “The Commandant’s Think Tank,” but was physically located in Quantico, Virginia, at the Marine Corps Combat Development Command (MCCDC) until summer 2013, when it moved to the Office of the Director of the Marine Corps Staff, located within the Pentagon. Unlike at MCCDC, where it was buried and had almost no contact with the commandant to influence his decision-making, after moving to the Pentagon it has had a much greater ability to red team policy decisions, including on the consideration to place women in ground combat positions; and Interviews with prior and current commandant red team members, and Marine officers, 2013 and 2014. See also, US Marine Corps, “Strategic Initiatives Group (SIG): ‘The Commandant’s Think Tank’,” accessed March 17, 2015 [www.hqmc.marines.mil/dmcs/Units/StrategicInitiativesgroup(SIG).aspx].

53. Interview with Lt. Col. Brian Ellis, November 25, 2014; and e-mail correspondence with Lt. Col. Brian Ellis, January 30, 2015.

54. Maj. Ron Rega spent three years as a Marine Corps red teamer, and completed a master’s thesis on the effectiveness of red teaming at Marine Corps University. He wrote that red teaming “requires a decision by the senior leadership of the organization on where to place the red team within the organization, what areas the red team will focus on, and how the red team will interact with the rest of the organization.” Rega, MEF and MEB Red Teams: Required Conditions and Placement Options, p. 43.

55. Melchor Antuñano, “Pilot Vision,” Federal Aviation Administration, 2002, p. 3.

56. Interview with Lt. Col. William Rasgorshek, November 17, 2014.

57. P.L. 106–398, Floyd D. Spence National Defense Authorization Act for Fiscal Year 2001, sec. 213, “Fiscal Year 2002 Joint Field Experiment,” October 30, 2000.

58. Roxana Tiron, “‘Millennium Challenge’ Will Test U.S. Military Jointness,” National Defense Magazine, August 2001, p. 20; Lt. Col. H.R. McMaster, “Crack in the Foundation: Defense Transformation and the Underlying Assumption of Dominant Knowledge in Future War,” US Army War College, November 2003; and Hearing of the Senate Armed Services Committee, Subcommittee on Emerging Threats and Capabilities, “Special Operations Military Capabilities, Operational Requirements, and Technology Acquisition in Review of the Defense Authorization Request for Fiscal Year 2003,” March 12, 2002.

59. Department of Defense, “Media Availability with Defense Secretary Rumsfeld and Norwegian MoD,” July 29, 2002.

60. Department of Defense, “General Kernan Briefs on Millennium Challenge 2002,” July 18, 2002.

61. Interview with Gen. Buck Kernan, June 24, 2014.

62. For information on the Running Start plan, see, Bob Woodward, Plan of Attack (New York: Simon and Schuster, 2004), p. 97.

63. Joint Warfighting Center, “Commander’s Handbook for an Effects-Based Approach to Joint Operations,” February 24, 2006, p. viii.

64. Interview with Lt. Gen. Paul Van Riper, May 31, 2013.

65. Department of Defense, “General Kernan Briefs on Millennium Challenge 2002,” July 18, 2002.

66. Thom Shanker, “Iran Encounter Grimly Echoes ’02 War Game,” New York Times, January 12, 2008, p. A1.

67. Interview with Lt. Gen. Paul Van Riper, May 23, 2014.

68. Interview with Gen. B. B. Bell, May 19, 2014.

69. Interview with Gen. Buck Kernan, June 24, 2014.

70. Ibid.

71. Interview with Lt. Gen. Paul Van Riper, May 31, 2013.

72. Ibid.

73. Interview with Lt. Gen. Paul Van Riper, May 23, 2014.

74. Sean D. Naylor, “Fixed War Games?” Army Times, August 26, 2002, p. 8; Van Riper later acknowledged, “I knew that e-mail would get into the media because the OPFOR guys were so ticked off.” Interview with Lt. Gen. Paul Van Riper, May 23, 2014.

75. Department of Defense, “Gen. Kernan and Maj. Gen. Cash Discuss Millennium Challenge’s Lessons Learned,” September 17, 2002.

76. Naylor, “Fixed War Games?”

77. Department of Defense, “Pentagon Briefing,” August 20, 2002.

78. US Joint Forces Command, “U.S. Joint Forces Command Millennium Challenge 2002: Experiment Report,” undated.

79. Ibid., p. F-11.

80. Department of Defense, Defense Science Board Task Force on the Role and Status of DoD Red Teaming Activities, September 2003, p. 18.

81. Sandra Erwin, “‘Persistent’ Intelligence Feeds Benefit Air Combat Planners,” National Defense Magazine, October 2002, pp. 20–21.

82. Interview with Gen. B. B. Bell, May 19, 2014.

83. Interview with Gen. Buck Kernan, June 24, 2014.

84. Testimony of Maj. Gen. Eli Zeira, Agranat Commission, 1974.

85. Barbara Opall-Rome, “40 Years Later: Conflicted Accounts of Yom Kippur War,” Defense News, October 6, 2013.

86. Testimony of Moshe Dayan, Agranat Commission, 1974.

87. Aryeh Shalev, Israel’s Intelligence Assessment Before the Yom Kippur War: Disentangling Deception and Distraction (Portland, OR: Sussex Academic Press, 2010), p. viii.

88. Government of Israel, “Agranat Commission,” 2008, accessed March 17, 2015 [www.knesset.gov.il/lexicon/eng/agranat_eng.htm].

89. Lt. Col. Shmuel, “The Imperative of Criticism,” Studies in Intelligence, 24, 1985, p. 65. This was originally printed in IDF Journal, 2(3), May 1985.

90. It is at times also translated as “Research Unit” or “Internal Audit Unit,” but is referred to by IDF officers and in internal documents as simply “control.”

91. Zach Rosenzweig, “‘The Devil’s Advocate’: The Functioning of the Oversight Department of [IDF] Military Intelligence,” trans. Uri Sadot, Israel Defense Forces, April 10, 2013.

92. E-mail correspondence with a former Israeli military official, June 25, 2014; Yosef Kuperwasser, “Lessons from Israel’s Intelligence Reforms,” Saban Center for Middle East Policy, Analysis Paper no. 14, Brookings Institution, October 2007, p. 4; and Interview with a former CIA official, November 13, 2014. Mahleket Bakara alternative analyses have also been shared with their US intelligence counterparts, but primarily after the issue is no longer relevant.

93. Interview with Bruce Riedel, November 13, 2014.

94. United Kingdom Ministry of Defence, Red Teaming Guide, second ed., January 2013, p. 4–2.

95. Air Chief Marshal, Sir Jock Stirrup, Chief of Defence Staff, “RUSI Christmas Lecture,” January 4, 2010.

96. Interview with Development, Concepts and Doctrine Centre red team members, April 20, 2015.

97. Interview with Brig. Tom Longland, November 25, 2014; and United Kingdom Ministry of Defence, Red Teaming Guide, pp. 1-4, 2-2.

98. Interview with Brig. Tom Longland, November 25, 2014; and interview with DCDC red team members, 2015.

99. United Kingdom Ministry of Defence, Red Teaming Guide, p. 2–2.

100. When establishing the unit at Norfolk, officials and staffers consciously rejected the terms “red team” and “red cell” in favor of “alternative analysis” in order to more clearly emphasize that “the capability provides and captures the importance of critical analysis over the adversarial mind-set implied by the name Red Team.” See, North Atlantic Treaty Organisation, “Bi-Stratetic Commands Concept for Alternative Analsysis (AltA),” April 23, 2012, p. 5; and interview with NATO official, May 22, 2015.

101. Interview with Johannes “Hans” de Nijs, June 20, 2014.

102. Interview with Lt. Gen. Phil Jones, June 20, 2014.

103. Ibid.

104. Interview with Col. Kevin Benson, May 21, 2012.

105. E-mail correspondence with Col. Kevin Benson, July 15, 2014.

106. Defense Manpower Data Center, Department of Defense Active Duty Military Personnel by Rank/Grade, accessed July 17, 2014; and Congressional Budget Office, Long-Term Implications of the 2013 Future Years Defense Program, July 2012, p. 14.

107. Interview with Lt. Gen. John Toolan, June 25, 2014.

108. Malcolm Gladwell, “Paul Van Riper’s Big Victory: Creating Structure for Spontaneity,” in Blink: The Power of Thinking Without Thinking (New York: Little Brown and Company, 2005), pp. 99–146.

CHAPTER THREE

1. Robert Gates, “The Prediction of Soviet Intentions,” Studies in Intelligence, 17(1), 1973, p. 46.

2. Office of the Director of National Intelligence, “DNI Releases Requested Budget Figure for FY 2016 Appropriations for the National Intelligence Program,” February 2, 2015; and Department of Defense, “DoD Releases Military Intelligence Program Base Request for Fiscal Year 2016,” February 2, 2015.

3. Richard Helms, A Look over My Shoulder: A Life in the Central Intelligence Agency (New York: Ballantine Books, 2003), p. 237.

4. Paul Pillar, Terrorism and U.S. Foreign Policy (Washington, DC: Brookings Institution Press, 2001), p. 114.

5. Most analytical products are supposed to be based upon formal requirements detailed in the National Intelligence Priorities Framework (NIPF), the mechanism by which the White House and the Office of the Director of National Intelligence—which oversees the other sixteen IC agencies—prioritizes intelligence collection and analysis tasks. Other analytical products are self-initiated by the analysts themselves with the approval of a supervisor, or are produced in an ad hoc manner in response to a pressing issue.

6. Interview with a senior intelligence community official, March 2014.

7. In March 2015, Director John Brennan announced a reorganization of the CIA. The final revamped structure had not been published as of June 2015. See, CIA, Unclassified Version of March 6, 2015, Message to the Workforce from CIA Director John Brennan, “Our Agency’s Blueprint for the Future,” March 6, 2015.

8. CIA, The Performance of the Intelligence Community Before the Arab-Israeli War of October 1973: A Preliminary Post-Mortem Report, December 1973, p. 22. In 1973, Director of Central Intelligence Richard Helms also mandated that the IC “develop regular systems to be implemented by the [National Intelligence Officers] to ensure that serious divergent points of view and conflicting elements of information not be submerged by managerial fiat or the mechanism of reinforcing consensus. . . . Such systems will also be charged with ensuring the establishment of means to provide the views of devils’ advocates, adversary procedures, and the use of gaining techniques as appropriate.” In practice, this recommendation was never implemented.

9. A January 2015 updated directive instructs: “Analysts must perform their functions with objectivity and with awareness of their own assumptions and reasoning. They must employ reasoning techniques and practical mechanisms that reveal and mitigate bias.” This type of formal guidance is impossible for analysts to take into account when drafting analytical products day-to-day, according to several dozen analysts interviewed for this book. See, Office of the Director of National Intelligence, Intelligence Community Directive 203, updated January 2, 2015, p. 2.

10. Interview with Andrew Liepman, July 23, 2014.

11. Interview with Carmen Medina, June 2, 2014.

12. Interview with Gregory Treverton, January 6, 2014.

13. Interview with Gen. Michael Hayden, April 30, 2014.

14. Interview with a senior intelligence community official, April 2014.

15. Interview with Michael Morell, April 16, 2014.

16. Hearing of the House Permanent Select Committee on Intelligence, “Worldwide Threat Hearing,” February 10, 2011.

17. Interviews with intelligence community analysts and officials, 2011–2014; and Paul Lehner, Avra Michelson, and Leonard Adelman, “Measuring the Forecast Accuracy of Intelligence Products,” Mitre Corporation, December 2010.

18. CIA, “Estimate of Status of Atomic Warfare in the USSR,” September 20, 1949, p. 1.

19. CIA, “Declassified National Intelligence Estimates on the Soviet Union and International Communism,” updated October 5, 2001, accessed March 17, 2015.

20. Albert Wohlstetter, “Is There a Strategic Arms Race?” Foreign Policy, ١٥, 1974, pp. 3–20; and Anne Hessing Cahn, Killing Détente: The Right Attacks the CIA (University Park, PA: Pennsylvania State University Press, 1998), pp. 11–13.

21. CIA, NIE 11–3/8–74, Soviet Forces for Intercontinental Conflict Through 1985, November 14, 1974, pp. 10–11.

22. White House, Memorandum of Conversation, August 8, 1975.

23. Memorandum for Secretary of Defense, Deputy Secretary of State, and Director of Central Intelligence “Trial Modification to the NIE Process,” undated.

24. Letter from the Director of Central Intelligence (Colby) to President Ford, November 21, 1975.

25. Letter from the Chairman of the President’s Foreign Intelligence Advisory Board (Cherne) to the Director of Central Intelligence (Bush), June 8, 1976.

26. George A. Carver, Note for the Director [of Central Intelligence], May 26, 1976.

27. Cahn, Killing Détente: The Right Attacks the CIA, p. 139.

28. Interview with Robert Gates, June 24, 2014.

29. Cahn, Killing Détente: The Right Attacks the CIA, p. 153.

30. Richard Pipes, “Team B: The Reality Behind the Myth,” Commentary, October 1986, pp. 25–40.

31. Interview with Maj. Gen. Jasper Welch, July 1, 2014. At the time, Welch was the Air Force’s assistant chief of staff for studies and analysis, where he had been leading a study into why the Soviet expenditures on air defenses were far beyond what they required. When he received a call from one of his Air Force bosses asking if he wanted to serve on the Team B, he replied, “Sure, I will work on the Air Defense panel.” However, he was then instructed, “No, you will be on the [Strategic Objectives Panel]. You have to do it.”

32. Cahn, Killing Détente: The Right Attacks the CIA, p. 159, citing interview with Adm. Daniel Murphy, November 9, 1989.

33. CIA, “Intelligence Community Experiment in Competitive Analysis: Soviet Strategic Objectives an Alternative View Report of Team B,” National Archives, December 1976.

34. Melvin Goodman, “Chapter 6,” in National Insecurity: The Cost of American Militarism (San Francisco, CA: City Lights Books, 2013).

35. Anne Hessing Cahn, who interviewed almost every participant in the Team B experiment for her masterful historical account, Killing Détente: The Right Attacks the CIA, recalled that all of its members were diametrically opposed to improved relations with the Soviet Union. Even speaking with them after the Cold War, she noted: “I could predict with one-hundred-percent accuracy their answer to most questions based upon their careers and ideological affiliation. They still hated the Soviet Union, and they distrusted the findings of the CIA.” Interview with Anne Hessing Cahn, June 2, 2014.

36. CIA, “Intelligence Community Experiment in Competitive Analysis: Soviet Strategic Objectives an Alternate View Report of Team B,” pp. 1 and 14.

37. Memorandum from the Director of Central Intelligence (Bush) to Recipients of National Intelligence Estimate 11-3/8-76, undated.

38. See, Murney Marde, “Carte to Inherit Intense Dispute on Soviet Intentions,” Washington Post, January 2, 1977, p. A1. See also, Cahn, Killing Détente: The Right Attacks the CIA, p. 179; and Ibid., p. 182, citing interview with Richard Pipes, August 15, 1990.

39. Senate Select Committee on Intelligence, Subcommittee on Collection, Production, and Quality, “The Nation’s Intelligence Estimates A-B Team Episode Concerning Soviet Strategic Capability and Objectives,” February 16, 1978.

40. Memorandum from the Director of Central Intelligence (Bush) to the Chairman of the President’s Foreign Intelligence Advisory Board (Cherne), January 19, 1977.

41. Interview with Maj. Gen. Jasper Welch, July 1, 2014.

42. Cahn, Killing Détente: The Right Attacks the CIA, p. 160.

43. Interview with Robert Gates, June 24, 2014.

44. Office of Rep. Pete Hoekstra, “Hoekstra Calls for Independent Red Team on Iran Nuclear Issue,” October 6, 2009. In fact, the 2007 NIE was red teamed because its key findings were so different from previous NIEs. This idea was refloated in April 2015 when Michael Mukasey, former US attorney general, and Kevin Carroll, former senior counsel to the House Committee on Homeland Security, called on “House and Senate leaders of both parties [to] ask former senior national-security officials to study raw intelligence-reporting on Iran, and direct the administration legislatively if necessary to give them the data needed to make an informed judgment. This ‘Team B’ should then report its findings periodically not only to the administration, but also to congressional leaders and the presidential nominees of both parties once they are chosen.” See, Michael Mukasey and Kevin Carroll, “The CIA Needs an Iran ‘Team B’,” Wall Street Journal, April 14, 2015, p. A13.

45. Richard Clarke, Against All Enemies: Inside America’s War on Terror (New York: Free Press, 2004), p. 184.

46. Interview with a former intelligence community official, May 2014.

47. National Commission on Terrorist Attacks upon the United States (herein 9/11 Comission), The 9/11 Commission Report: The Attack from Planning to Aftermath, 2004, p. 117.

48. Ibid., p. 116.

49. Interview with Bruce Riedel, January 23, 2007.

50. Bill Clinton, My Life (New York: Knopf, 2004), p. 803.

51. Interview with John Lauder, director of the Nonproliferation Center at the time of Al Shifa, June 20, 2014.

52. Interview with Jami Miscik, June 9, 2014.

53. Interview with Mary McCarthy, May 15, 2014.

54. Interview with Phyllis Oakley, April 2014; and James Risen, “To Bomb Sudan Plant, or Not: A Year Later, Debates Rankle,” New York Times, October 29, 1999, p. A1.

55. Interview with a former intelligence community official, May 2014; and Vernon Loeb, “U.S. Wasn’t Sure Plant Had Nerve Gas Role; Before Sudan Strike, CIA Urged More Tests,” Washington Post, August 21, 1999, p. A01.

56. Risen, “To Bomb Sudan Plant, or Not: A Year Later, Debates Rankle,” p. A1.

57. Interview with Paul Pillar, deputy chief of central intelligence’s Counterterrorist Center at the time of Al Shifa, September 2006; and Interviews with small group members, 2013–2014.

58. Interview with Gen. Anthony Zinni, February 2008.

59. Shelton recalled that, after the attack on Al Shifa, “the intel started to fade on us, and it turned out that this CIA intelligence had not really been collected at the pharmaceutical plant, but rather three hundred yards away from it. And now—by the way—the quarter teaspoon of soil sample turned out to have been collected two years earlier.” See, Gen. Hugh Shelton with Ronald Levinson and Malcolm McConnell, Without Hesitation: The Odyssey of an American Warrior (New York: St. Martin’s Press, 2010), p. 350.

60. Interview with a former White House official, May 2014.

61. Daniel Pearl, “New Doubts Surface over Claims That Plant Produced Nerve Gas,” Wall Street Journal, August 28, 1998.

62. George Tenet, with Bill Harlow, At the Center of the Storm: My Years at the CIA (New York: HarperCollins, 2007), p. 117.

63. Statement of William S. Cohen to the National Commission on Terrorist Attacks Upon the United States, March 23, 2004, p. 14.

64. Interview with Jami Miscik, June 9, 2014.

65. Interview with Thomas Pickering, April 21, 2014.

66. The following section is based primarily upon interviews with current and former CIA and intelligence community staffers and officials, other government officials, and Tenet, with Harlow, At the Center of the Storm, pp. 194–195.

67. Ibid., p. 185.

68. Interview with Gen. David Petraeus, February 19, 2014.

69. Interview with Carmen Medina, June 2, 2014.

70. Interview with Jami Miscik, May 21, 2012.

71. Interview with Paul Frandano, June 18, 2013.

72. Interview with Philip Mudd, April 2014; Rodney Faraon, an analyst in Tenet’s office in the CIA Red Cell’s earliest days, recalled of the unit’s three-page memos: “Some of it worked, and some of it was kind of stupid, but all of it got read.” Interview with Rodney Faraon, May 27, 2014.

73. Interview with Paul Frandano, June 18, 2013.

74. Interview with Gen. Michael Hayden, January 21, 2014.

75. Interview with Col. James Baker, January 14, 2014.

76a. CIA Red Cell Memorandum, “Afghanistan: Sustaining West European Support for the NATO-led Mission,” March 11, 2010, released by Wikileaks, March 26, 2010.

76. P.L. 108–458, Intelligence Reform and Terrorism Prevention Act of 2004, sec. 1017, “Alternative Analysis of Intelligence by the Intelligence Community,” US Congress, December 17, 2004.

77. Interview with Gen. David Petraeus, February 29, 2014.

78. Interview with Stephen Hadley, June 12, 2014.

79. Full disclosure: The author is a columnist for Foreign Policy.

80. Interview with Robert Gates, June 24, 2014.

81. Interview with a senior intelligence community official, February 2014.

82. Interview with Gen. Michael Hayden, January 21, 2014.

83. Interview with Stephen Hadley, June 12, 2014.

84. Interview with a CIA Red Cell member, March 26, 2014.

85. Interview with Michael Morell, April 16, 2014.

86. Joby Warrick, The Triple Agent: The Al-Qaeda Mole Who Infiltrated the CIA (New York: Vintage Books, 2011), p. 206.

87. Gates, Duty: Memoirs of a Secretary at War, p. 539.

88. Interviews with intelligence community officials, March and April 2014; and Mark Owen, with Kevin Maurer, No Easy Day: The Firsthand Account of the Mission that Killed Osama Bin Laden (New York: Penguin, 2012), pp. 15–26.

89. Michael Morell, with Bill Harlow, The Great War of Our Time: The CIA’s Fight Against Terrorism From Al Qa’ida to ISIS (New York: Twelve, 2015), p. 160.

90. Tim Starks, “Femstein: Tip on Bin Laden May Not Have Come from Harsh Interrogations,” Congressional Quarterly Today, May 3, 2011.

91. For information about D’Andrea see, Greg Miller, “At CIA, a Convert to Islam Leads the Terrorism Hunt,” Washington Post, March 24, 2012; and Mark Mazzetti and Matt Apuzzo, “Deep Support in Washington for C.I.A.’s Drone Missions,” New York Times, April 26, 2015, p. A1.

92. Interview with Michael Morell, April 2014.

93. Mark Bowden, The Finish: The Killing of Osama Bin Laden (New York: Grove Press, 2012), p. 163; and Seth G. Jones, Hunting in the Shadows: The Pursuit of Al Qa’ida Since 9/11 (New York: W.W. Norton, 2012), p. 424.

94. Morell, with Harlow, The Great War of Our Time: The CIA’s Fight Against Terrorism from Al Qa’ida to ISIS, p. 160.

95. Interview with Michael Leiter, January 21, 2014.

96. Interview with a senior White House official, April 2014.

97. Interview with Michael Leiter, January 21, 2014.

98. Interview with Andrew Liepman, July 23, 2014.

99. Interview with Michael Morell, April 16, 2014.

100. Jeffrey Friedman and Richard Zeckhauser, “Handling and Mishandling Estimative Probability: Likelihood, Confidence, and the Search for Bin Laden,” Intelligence and National Security, May 2014, pp. 12 and 20.

101. Interview with Michael Leiter, January 21, 2014; and Interview with Andrew Liepman, July 23, 2014.

102. Bergen, Manhunt: The Ten-Year Search for Bin Laden from 9/11 to Abbottabad, p. 196.

103. Bowden, The Finish: The Killing of Osama Bin Laden, p. 161. Three days after the operation, Obama told 60 Minutes correspondent Steve Kroft in an interview: “At the end of the day, this was still a 55/45 situation. I mean, we could not say definitively that bin Laden was there.”

104. Obama’s default to a fifty-fifty estimate is how people tend to deal with conflicting, complex information. See, Baruch Fischhoff and Wändi Bruine de Bruin, “Fifty-Fifty = 50%?,” Journal of Behavioral Decisionmaking (2), 1999, pp. 149–163.

105. Interviews with senior intelligence community officials, March–April 2014; and Leon Panetta, Worthy Fights (New York, Penguin Press, 2014), pp. 314–315.

106. Interview with Michael Morell, April 16, 2014; and Interview with a senior administration official, February 2014. Morell’s own estimate was 60 percent, but he still recommended the SEAL raid given that eliminating bin Laden was such an important objective. See, Morell, with Harlow, The Great War of Our Time: The CIA’s Fight Against Terrorism from Al Qa’ida to ISIS, p. 161.

107. Interview with Robert Gates, June 24, 2014.

108. Bergen, Manhunt: The Ten-Year Search for Bin Laden from 9/11 to Abbottabad, p. 196.

109. Interview with Michael Leiter, January 21, 2014.

110. Interview with a senior administration official, February 2014.

111. Interview with Andrew Liepman, July 23, 2014.

112. Interview with Robert Gates, June 24, 2014.

113. Interview with Stephen Hadley, June 12, 2014.

114. Interview with a senior White House official, April 2014.

CHAPTER FOUR

1. Interview with Bogdan Dzakovic, June 11, 2013.

2. Interview with Stephen Sloan, July 9, 2014; and Stephen Sloan, “Almost Present at the Creation: A Personal Perspective of a Continuing Journey,” Journal of Conflict Studies, 24(1), 2004, pp. 120–134.

3. Sloan’s dissertation was titled, “An Examination of Lucian W. Pye’s Theory of Political Development: Through a Case Study of the Indonesian Coup of 1965” (University of Michigan–Ann Arbor, 1967).

4. The series began on July 28, 1974, with the article: “Israelis Live with Tensions.”

5. Stephen Sloan, “‘International Terrorism’ Being Taught in OU Classroom,” ADA Evening News, May 5, 1977, p. 7C. In this article, Sloan presciently warned: “It’s one thing for a band of insurgents to knock off a rural official, but it’s quite another when a small group can knock out the electronic grid of a large modern city.”

6. The pseudonym “Leila” was a reference to Leila Khaled, a female member of the Popular Front for the Liberation of Palestine and well-known airline hijacker. She is famous for her role in the August 1969 hijacking of TWA Flight 840 and the Dawson’s Field hijackings during Black September in Jordan in 1970. The quote is from an interview with Stephen Sloan, July 9, 2014.

7. Ibid.; ironically, the exact same office building from which federal and international law enforcement officials observed the simulated hostage-taking was where Zacarias Moussaoui, an Al Qaeda member intercepted by the FBI less than a month before 9/11 and later convicted for conspiracy to kill Americans, took simulated flight training classes a quarter-century later. Ihab Ali Nawawi, Osama bin Laden’s personal pilot in the 1990s, also took lessons at the school.

8. US Department of State, Office of Combating Terrorism, Terrorist Skyjackings: A Statistical Overview of Terrorist Skyjackings from January 1968 Through June 1982, 1982.

9. Six-part series in The Oklahoman, July 28, 1974, September 30, 1974, October 2–4, 1974; six-part series in The Oklahoman, November 12–19, 1975; Stephen Sloan and Richard Kearney, “An Analysis of a Simulated Terrorist Incident,” The Police Chief, June 1977, pp. 57–59; and Stephen Sloan, “Stimulating Terrorism: From Operational Techniques to Questions of Policy,” International Studies Notes, 5(4), 1978.

10. Stephen Sloan, “Almost Present at the Creation: A Personal Perspective of a Continuing Journey,” The Journal of Conflict Studies, 24(1), 2004.

11. Stephen Sloan and Robert Bunker, Red Teams and Counterterrorism Training (Norman, OK: University of Oklahoma Press, 2011), pp. 91–101.

12. Interview with a US Department of Homeland Security (DHS) official, March 12, 2014; and DHS, U.S. Department of Homeland Security Annual Performance Report: Fiscal Years 2014–2016, February 2, 2015, p. 119.

13. Interview with a DHS official, March 12, 2014.

14. Jason Miller, “DHS Teams Hunt for Weaknesses in Federal Cyber Networks,” Federal News Radio, July 11, 2012.

15. Hearing of the Senate Committee on Commerce, Science, and Transportation, “Are Our Nation’s Ports Secure? Examining the Transportation Worker Identification Credential Program,” May 10, 2011.

16. Interview with Wayne McElrath, director of the Government Accountability Office’s (GAO’s) Office of Special Investigation, August 23, 2013; the standards for GAO and all government vulnerability probes can be found in the “yellow book.” See GAO, “Government Auditing Standards,” December 2011.

17. GAO, “Border Security: Summary of Covert Tests and Security Assessments for the Senate Committee on Finance, 2003–2007,” May 2008, p. 3.

18. GAO, “Border Security: Additional Steps Needed to Ensure that Officers Are Fully Trained,” December 2011, p. 4.

19. GAO, “Border Security: Summary of Covert Tests and Security Assessments for the Senate Committee on Finance, 2003–2007,” May 2008, pp. 8–12.

20. GAO, “Combating Nuclear Smuggling: Risk-Informed Cover Assessments and Oversight of Corrective Actions Could Strengthen Capabilities at the Border,” September 2014, pp. 14–15.

21. GAO, “Border Security: Additional Steps Needed to Ensure That Officers Are Fully Trained,” December 2011 [www.gao.gov/products/GAO-12–269]. See, “Recommendations.”

22. Ibid., pp. 2 and 10.

23. Mark Holt and Anthony Andrews, “Nuclear Power Plant Security and Vulnerabilities,” Congressional Research Service, January 3, 2014, p. 9.

24. Christine Cordner, “PG&E Offers More Details on Substation Attack, Tallies Up Recovery Cost at over $15M,” SNL Federal Energy Regulatory Commission, June 25, 2014.

25. Richard Serrano and Evan Halper, “Sophisticated but Low-tech Power Grid Attack Baffles Authorities,” Los Angeles Times, February 11, 2014, p. A1.

26. Pacific Gas and Electric, “PG&E Announces Request for Information on Metcalf Substation Attack,” April 10, 2014.

27. David Baker, “Thieves Raid PG&E Substation Hit by Snipers in 2013,” Sfgate.com, August 27, 2014.

28. Rebecca Smith, “Assault on California Power Station Raises Alarm on Potential for Terrorism,” Wall Street Journal, February 5, 2014, p. A1.

29. Cordner, “PG&E Offers More Details on Substation Attack, Tallies Up Recovery Cost at over $15M.”

30. Rebecca Smith, “Federal Government Is Urged to Prevent Grid Attacks,” Wall Street Journal, July 6, 2014.

31. Interview with Steve Elson, June 12, 2013.

32. Report of the President’s Commission on Aviation Security and Terrorism, May 15, 1990.

33. Ibid., p. ii.

34. P.L. 104–64, Federal Aviation Reauthorization Act of 1996, sec. 312, “Enhanced Security Programs,” October 9, 1996; and 9/11 Commission, Memorandum for the Record, Interview with Bruce Butterworth, former Director for Policy and Planning at the FAA, September 29, 2003, p. 5.

35. The associate administrator for Civil Aviation Security from 1993 to 2000 was Rear Adm. Cathal Flynn. When asked about his role and responsibility in overseeing the FAA red team, he puzzlingly replied: “We never used a Red Team in FAA Security.” This is demonstrably false, but deciding not to remember the FAA Red Team might indicate what little impact it had within the CAS. See e-mail correspondence with Rear Adm. Cathal Flynn, May 20, 2014.

36. GAO, “Aviation Safety: Weaknesses in Inspection and Enforcement Limit FAA in Identifying and Responding to Risks,” February 1998, pp. 7–8, 24, and 61–62.

37. Flynn told 9/11 Commission investigators that “red team testing was made ‘easy’ because it would help the FAA to obtain a civil penalty against the airline if the failure were obvious and glaring.” Despite finding many obvious and glaring errors, there is no record of a red team finding ever resulting in such a civil penalty. 9/11 Commission, Memorandum for the Record, “Interview with Rear Admiral Cathal ‘Irish’ Flynn, USN (ret),” September 9, 2003.

38. Interviews with Steve Elson, June 12, 2013, and June 11, 2014.

39. US Department of Transportation (DOT), Office of Inspector General, Semiannual Report to the Congress, October 1, 1999–March 31, 2000, p. 17.

40. Letter from [Special Counsel] Elaine Kaplan to the President, “Re: OSC File No. DI-02–0207,” March 18, 2003, p. 4.

41. For example, the FAA Administrator from 1993 to 1996 was David Hinson, cofounder of Midway Airlines. He was replaced from 1996 to 1997 by Linda Daschle, who previously was the chief lobbyist for the Air Transport Association, the airline industry’s main lobby. She was replaced from 1997 to 2002 by Jane Garvey, who had been the director of Logan International Airport. See, Public Citizen, Delay, Dilute and Discard: How the Aviation Industry and the FAA Have Stymied Aviation Security Recommendations, October 2001; and Doug Ireland, “I’m Linda, Fly Me,” LA Weekly, January 16, 2003.

42. Jim Morris, “Since Pan Am 103 a ‘Façade of Security’,” U.S. News & World Report, 130 (7), February 19, 2001, p. 28.

43. GAO, Aviation Security: Long-Standing Problems Impair Airport Screeners’ Performance, June 2000, p. 7. In 1997, the FAA declared that the results of airport screeners’ performances would henceforth be sensitive security information, and could therefore not be released.

44. Deborah Sherman, Investigative Report, Fox 25, May 6, 2001. The report was delivered along with a letter written by Sullivan that warned prophetically: “With the concept of jihad, do you think it would be difficult for a determined terrorist to get on a plane and destroy himself and all other passengers? . . . Think what the result would be of a coordinated attack that took down several domestic flights on the same day. The problem is that with our current screening system, this is more than possible. Given time, with current threats, it is almost likely.” John Kerry’s office sent the entire package that Dzakovic delivered to the DOT inspector general, where it was received by the same officials who had repeatedly heard these concerns from Dzakovic directly, but had decided not to investigate them.

45. The 9/11 Commission Report: The Attack from Planning to Aftermath, pp. 242–245.

46. The 9/11 Commission would not be formed for another thirteen months. After strongly opposing any meaningful investigation into the most lethal and costly terrorist attack in American history, President Bush reversed course in November 2002, and announced the forming of the Commission with Henry Kissinger as its chair.

47. Office of Special Counsel, “U.S. Office of Special Counsel Sends Report Confirming Gross Management of FAA’s Red Team, Resulting in Substantial and Specific Danger to Public Safety,” March 18, 2003.

48. Hearing of the House Committee on Homeland Security, Subcommittee on Transportation Security, “Examining TSA’s Cadre of Criminal Investigators,” January 28, 2014; Hearing of the House Homeland Security Committee Transportation Security Subcommittee, “Transportation Security Administration’s Efforts to Advance Risk-Based Security,” March 14, 2013; and Interview with Bogdan Dzakovic, June 11, 2013. These covert smuggling attempts were incorrectly reported by the media as “red team” tests. In fact, they were done by auditors without “any specialized background or training,” including accountants. See, Hearing of the Senate Committee on Homeland Security and Governmental Affairs, “Transportation Security Administration Oversight,” June 9, 2015. As a direct result of these well-publicized security shortcomings, the acting administrator for the TSA, Melvin Carraway, was reassigned within DHS.

49. “Press Release: Enhanced Security Measures at Certain Airports Overseas,” US Department of Homeland Security, Transportation Security Administration, July 6, 2014.

50. Evan Booth, “Terminal Cornucopia,” presentation at SkyDogCON 2013, Nashville, TN, October 26, 2013, accessed March 17, 2015 [www.youtube.com/watch?v=PiGK2rk5524].

51. Hearing of the Senate Committee on Appropriations and Senate Committee on the Budget, Subcommittee on Transportation, “Federal Aviation Administration: Challenges in Modernizing the Agency,” February 3, 2000.

52. 9/11 Commission, Memorandum for the Record, “Interview with Bruce Butterworth, former Director for Policy and Planning at the FAA,” September 29, 2003, p. 6.

53. Federation of American Scientists, The Menace of MANPADS, 2003.

54. Colin Powell, Comments to Asia-Pacific Economic Cooperation Forum, Bangkok, Thailand, October 18, 2003.

55. P. L. 108–458, Intelligence Reform and Terrorism Prevention Act of 2004, US Congress, December 17, 2004.

56. GAO, Aviation Security: A National Strategy and Other Actions Would Strengthen TSA’s Efforts to Secure Commercial Airport Perimeters and Access Controls, September 2009, p. 21.

57. James Chow et al., Protecting Commercial Aviation Against the Shoulder-Fired Missile Threat (Santa Monica, CA: RAND Corporation, 2005), p. 15.

58. Paul May, “Going Gaga for Online Radio,” Guardian, January 8, 2003, p. 5.

59. US Department of State, Bureau of Political-Military Affairs, “MANPADS: Combatting the Threat to Global Aviation from Man-Portable Air Defense Systems,” July 27, 2011.

60. Office of the Director of National Intelligence, Press Briefing with Intelligence Officials, July 22, 2014.

61. Kirk Semple and Eric Schmitt, “Missiles of ISIS May Pose Peril for Aircrews in Iraq,” New York Times, October 27, 2014, p. A1.

62. John Pistole, “TSA: Toward a Risk-Based Approach to Aviation Security,” presentation at the Aspen Security Forum, Aspen, CO, July 23, 2014; and Rory Jones, Robert Wall, and Orr Hirschauge, “Attacks Spur Debate on Antimissile Systems for Passenger Jets,” Wall Street Journal, July 24, 2014, p. A8.

63. Cathy Scott-Clark and Adrian Levy, The Siege: 68 Hours Inside the Taj Hotel (New York: Penguin Books, 2013); and Angela Rabasa et al., “The Lessons of Mumbai,” Occasional Paper, RAND Corporation, January 2009.

64. NYPD Intelligence Division, “Mumbai Attack Analysis” (Law Enforcement Sensitive Information as of December 4, 2008).

65. Interviews with Commissioner Ray Kelly, January 2014; and Hearing of the Senate Committee on Homeland Security and Governmental Affairs, “Lessons from the Mumbai Terrorist Attacks,” January 8, 2009.

66. Interview with Capt. James Waters and “Bob,” March 31, 2014.

67. Interviews with Commissioner Ray Kelly, January 2014.

68. Interview with Capt. James Waters and “Bob,” March 31, 2014.

69. Star Trek II: The Wrath of Khan, directed by Nicholas Meyer (Paramount Pictures, 1982).

70. Interview with Mitchell Silber, March 6, 2014.

71. Interviews with NYPD officials, January–March 2014.

72. In January 2015, this arrangement was formalized with the establishment of the Capital Strategic Response Group, which increased the number of officers dedicated specifically to responding to multiple gunman terrorist attacks, like those that had occurred against Charlie Hebdo magazine three weeks earlier. See, “Police Commissioner Bratton’s Remarks at the ‘State of the NYPD’,” Police Foundation, January 29, 2015.

73. Ibid. As a result of the tabletop exercise, the NYPD also developed and implemented new highly secret methods for the pinpoint jamming of terrorists’ cellphone communications in crisis situations.

74. Ibid.; Patrice O’Shaughnessy, “NYPD Learns from Mumbai Terrorist Attack that Killed 174,” New York Daily News, February 15, 2009, p. 16.

75. Sean Gardiner, “NYPD Trains for New Type of Attack,” Wall Street Journal, December 20, 2010, p. A21.

76. See, “Raymond Parks,” LinkedIn, accessed March 17, 2015 [www.linkedin.com/pub/raymond-parks/6/566/a75].

77. Interviews with Raymond Parks, June 2014.

78. US Air Force, Air Force System Safety Handbook, Air Force Safety Agency, July 2000, p. 121.

79. iMPERVA, “Red Teaming, an Interview with Ray Parks of Sandia National Labs (SNL),” 2009.

80. Kevin Robinson-Avila, “Sandia Shows Off New Testing Complex,” Albuquerque Journal, May 9, 2014; and Hearing of the House Armed Services Committee, “Nuclear Weapons Modernization Programs: Military, Technical, and Political Requirements for the B61 Life Extensions Program and Future Stockpile Strategy,” October 29, 2013.

81. Interviews with Michael Skroch, June–July 2014.

82. Interview with Samuel Varnado, July 15, 2014.

83. Sandia National Laboratories, “Assessment Choices: When Choosing Sandia Makes Sense,” undated.

84. In practice, once a statement of work had been agreed upon between the government sponsor and IDART, National Nuclear Security Administration always signed off on the red team engagement. Technically, the Sandia Corporation, a subsidiary of Lockheed Martin Corporation, is contracted by the NNSA to manage and operate Sandia National Laboratories, for which they earn about $27 million annually, which includes a small fee from each IDART engagement. Dan Mayfield, “New Lockheed Sandia Contract Finalized Today,” Albuquerque Business First, April 30, 2014.

85. SCADA is used interchangeably with Industrial Control Systems (ICS). According to former National Security Agency (NSA) hacker Bob Stasio, who worked with IDART while at the NSA, IDART has a reputation for being especially effective at security assessments of critical infrastructure information systems. Interview with Stasio, June 30, 2014; on the growth of malicious SCADA attacks, see, Department of Homeland Security (DHS), National Cybersecurity and Communications Integration Center, “Internet Accessible Control Systems At Risk,” ICS-CERT Monitor, January–April 2014.

86. Interview with Samuel Varnado, July 15, 2014.

87. The program aired on April 24, 2003. Skroch noted that he wished he had said, “Duh! You think a national laboratory cannot impact an infrastructure? That’s not the important question. What is important is understanding what kind of adversary can!” Interview with Michael Skroch, June-July 2014.

88. According to IDART members, what was particularly notable about Invicta was that it was owned by Victor Sheymov, who ran the Soviet Union’s version of the NSA before he defected to the United States in 1980, and was staffed by former NSA hackers.

89. GAO, “Supply Chain Security: DHS Should Test and Evaluate Container Security Technologies Consistent with All Identified Operational Scenarios to Ensure the Technologies Will Function as Intended,” September 2010, p. 3; and Mark Greaves, “Ultralog Survivable Logistics Information Systems,” PowerPoint presentation, Defense Advanced Research Projects Agency, September 2002, slide 37.

90. Sandia National Laboratories, “Keep Telling Yourself: ‘The Red Team Is My Friend . . . ’,” 2000.

91. Interview with Dino Dai Zovi, July 18, 2014.

92. E-mail correspondence with Michael Skroch, July 17, 2014.

93. Sandia National Laboratories, “Red Teaming for Program Managers,” accessed March 17, 2015 [www.idart.sandia.gov/methodology/RT4PM.html].

94. Interview with Michael Skroch, June 4, 2014.

95. Mark Mateski, who was with the Sandia unit from 2005 to 2008, observed that IDART specialized in developing a methodology that was process-oriented and easily understood by non-specialists. The drawback to this commodification was that it did not allow for creativity or stratagem in conducting vulnerability probes. Interview with Mark Mateski, July 25, 2014.

96. Interview with IDART members, May–July 2014.

97. Hearing of the House Committee on Oversight and Government Reform, “Addressing Concerns about the Integrity of the U.S. Department of Labor’s Jobs Reporting,” June 6, 2012.

98. Scott Maruoka, CleanSweep Red Team Report, Sandia Report SAND2011, Sandia Laboratories Information Design Assurance Red Team, August 2011, p. 9.

99. Ibid., p. 11.

100. Denny Gulino, “US Labor Department Told ‘Adversaries’ Could Steal Data,” Market News International, July 11, 2012.

101. Scott Maruoka, “CleanSweep Mitigation Measures Acceptance Testing,” Sandia Laboratories Information Design Assurance Red Team, November 2012.

102. Department of Defense, Joint Service Chemical and Biological Defense Program, FY00–02 Overview, September 2001, p. 64.

103. Interview with Samuel Varnado, July 15, 2014.

CHAPTER FIVE

1. Dan Verton, “Companies Aim to Build Security Awareness,” Computerworld, November 27, 2000, p. 24.

2. Of course, executives and employees involved in red-teaming exercises sign internal corporate nondisclosure agreements as well.

3. US Census Bureau, Center for Economic Studies, “Business Dynamics Statistics 1976–2012,” updated 2012; and US Department of Labor Bureau of Labor Statistics, “Business Employment Dynamics: Establishment Age and Survival Data,” updated November 19, 2014.

4. Business Wire, “Lex Machina Releases First-Ever Patent Litigation Damages Report,” June 25, 2014.

5. H. Lee Murphy, “Saving More by Using Less: Efficiency Investments Can Pay Off over Time,” Crain’s Chicago Business, vol. 35, March 26, 2012, p. 23; and Sieben Energy Associates, “Strategic Consulting,” accessed March 17, 2015 [www.siebenenergy.com/services/strategicconsulting.aspx].

6. BAE Systems, “Testing and Lab Services,” 2014, accessed March 17, 2015 [www.baesystems.com/solutions-rai/cyber-security/cyber-security-solutions/penetration-testing].

7. John Gilbert, “Cyber Security ‘A Must’ for Telcos, Banking Institutions,” Malaysian Reserve, April 21, 2014.

8. PR Newswire, “360 Advanced Warns About Insider Threats: Is Your Data Already Out There and You Don’t Know It?” June 10, 2014.

9. Ram Shivakumar, “How to Tell Which Decisions Are Strategic,” California Management Review, 56(3), 2014, pp. 78–97.

10. International Business Machines, Chief Executive Office Study, 2010, p. 54.

11. Henry Mintzberg, The Rise and Fall of Strategic Planning: Reconceiving Roles for Planning, Plans, Planners (New York: The Free Press, 1984); Kees van der Heijden, Scenarios: The Art of Strategic Conversation, second ed. (West Sussex, UK: John Wiley and Sons, 2005); and Thomas Chermack, Scenario Planning in Organizations: How to Create, Use, and Assess Scenarios (San Francisco, CA: Berrett-Kohler Publishers, 2011).

12. James March and Herbert Simon, Organizations (New York: John Wiley and Sons, 1958), p. 185. Gresham’s Law is a monetary principle that explains what occurs when a new coin is given the same face value as an older coin that contains a greater amount of precious metal. The old coin will disappear from circulation as people begin to collect it because the value of the coin as a metal is now greater than its face currency value. See, “Gresham’s law,” Merriam Webster Dictionary, accessed March 17, 2015 [www.merriam-webster.com/dictionary/gresham’s%20law].

13. William Tolbert, The Power of Balance: Transforming Self, Society, and Scientific Inquiry (London, UK: Sage, 1991).

14. Paul Carroll and Chunka Mui, Billion Dollar Lessons: What You Can Learn from the Most Inexcusable Business Failures of the Last 25 Years (New York: Penguin Putnam, 2009), p. 234.

15. Interview with Jami Miscik, June 9, 2014.

16. This assumes that employees have the space to think and identify problems. A 2014 poll of 7,000 employees in eleven countries found that just 56 percent of American workers said they had regular time for creative thinking, and just 52 percent feel their environment enables creative thinking. See, Jack Morton Worldwide, “Creativity: How Business Gets to Eureka!” June 2014.

17. Interview with Ethan Burris, June 20, 2014.

18. Darcy Steeg Morris, Cornell National Social Survey 2009 (Ithaca, NY: Cornell University Survey Research Institute, 2009).

19. Ethan Burris, “The Risks and Rewards of Speaking Up: Managerial Reponses to Employee Voice,” Academy of Management Journal, 55(4), 2012, pp. 851–875; Compounding this problem, managers who perceive themselves as less competent are more likely to avoid or minimize improvement ideas from employees, as they challenge the manager’s already-threatened ego. See, Nathanael Fast, Ethan Burris, and Caroline Bartel, “Managing to Stay in the Dark: Managerial Self-efficacy, Ego Defensiveness, and the Aversion to Employee Voice,” Academy of Management Journal, 57(4), August 2014, pp. 1013–1034.

20. James Detert, Ethan Burris, David Harrison, and Sean Martin, “Voice Flows to and Around Leaders: Understanding When Units Are Helped or Hurt by Employee Voice,” Administrative Science Quarterly, 58(4), 2013, pp. 624–668.

21. Interview with Ethan Burris, June 20, 2014.

22. James Detert and Amy Edmondson, “Everyday Failures in Organizational Learning: Explaining the High Threshold for Speaking Up at Work,” Working Paper, Harvard Business School, October 2006.

23. Ibid., p. 3.

24. Carroll and Mui, Billion Dollar Lessons, pp. 277–291.

25. Ibid., p. 3.

26. Renee Dye, Olivier Sibony, and Vincent Truong, “Flaws in Strategic Decision Making,” McKinsey & Company, January 2009.

27. Though “business war game” is widely used to describe the activity, some consultants will use other terms if their clients are uncomfortable with the military connotation, such as “strategy review.” Some executives confuse business war games with competitive-intelligence exercises. The latter gathers and analyzes granular information about other firms, while the former applies the available information to help a firm reach the best strategic decision. The linking of warfare and business strategies dates to the earliest days of game theory. See, John McDonald, Strategy in Poker, Business, and War (New York: W.W. Norton, 1950).

28. Interview with Ken Sawka, May 9, 2014.

29. For example, in 2013, among the world’s 2,500 largest companies, 76 percent of all new CEOs were promoted from within. See, Strategy& and PricewaterhouseCoopers, The 2013 Chief Executive Study: Women CEOs of the Last 10 Years, April 2014, p. 3.

30. Sydney Finkelstein, Why Smart Executives Fail: And What You Can Learn from Their Mistakes (New York: Portfolio, 2003).

31. Interview with a financial-services sector senior vice president, June 28, 2014.

32. Interviews with Mark Chussil, June–July 2014.

33. Ibid.

34. Ibid.

35. Benjamin Gilad, Business War Games: How Large, Small, and New Companies Can Vastly Improve Their Strategies and Outmaneuver the Competition (Pompton Plains, NJ: Career Press, 2008).

36. Ben Gilad, war-gaming class, Fuld, Gilad, & Herring Academy of Competitive Intelligence, Cambridge, MA, June 16, 2014.

37. Interview with Ben Gilad, December 20, 2013.

38. Michael Porter, Competitive Strategy: Techniques for Analyzing Industries and Competitors (New York: Free Press, 1980); Demonstrating the inherent difficulty of beating the market through designing better strategies, in November 2012, Porter’s own strategy consulting firm, the Monitor Group, filed for bankruptcy protection before they were bought out by Deloitte. See, “Monitor’s End,” The Economist, November 14, 2012 [www.economist.com/blogs/schumpeter/2012/11/consulting].

39. Ben Gilad, war-gaming class, June 16, 2014.

40. Ibid.

41. Interview with Ben Gilad, December 20, 2013.

42. IBM Institute for Business Value, Capitalizing on Complexity: Insights from the Global Chief Executive Officer Study, 2010.

43. Kapersky Lab, IT Security Risks Survey 2014: A Business Approach to Managing Data Security Threats, 2014, p. 18.

44. Ponemon Institute, 2014 Cost of Cyber Crime Study: United States, sponsored by HP Enterprise Security, October 2014, p. 3.

45. Ibid.; and Verizon, 2015 Data Breach Investigations Report, April 2015, p. 4.

46. Symantec, Internet Security Threat Report, vol. 20, April 2015, pp. 7, 14. In a 2013 survey of small business owners, 44 percent of respondents said that they were the victim of a cyber attack, with the average associated costs being $8,700. See, National Small Business Association, 2013 Small Business Technology Survey, September 2013, p. 10.

47. Neiman Marcus Group, statement by Karen Katz, January 22, 2014.

48. Neiman Marcus Group, “Neiman Marcus Group LTD LLC Reports Second Quarter Results,” February 28, 2014, p. 9.

49. Becky Yerak, “Schnucks Calculates Potential Breach Hit,” Chicago Tribune, May 24, 2013, p. C1.

50. Target, “Target Reports Fourth Quarter and Full-Year 2013 Earnings,” February 26, 2014; and Rachel Abrams, “Target Puts Data Breach Costs at $148 Million, and Forecasts Profit Drop,” New York Times, August 5, 2014.

51. Market Research Media, “U.S. Federal Cybersecurity Market Forecast 2015–2020,” May 4, 2014, accessed May 21, 2015 [www.marketresearchmedia.com/?p=206].

52. Gartner, “Gartner Says Worldwide Information Security Spending Will Grow Almost 8 Percent in 2014 as Organizations Become More Threat-Aware,” August 22, 2014; and Gartner, “The Future of Global Information Security,” 2013.

53. Dave Evans, The Internet of Things: How the Next Evolution of the Internet Is Changing Everything, Cisco, April 2011, p. 3; and “Home, Hacked Home,” The Economist, July 12, 2014, p. SS14.

54. Daniel Halperin et al., “Pacemakers and Implantable Cardiac Defibrillators: Software Radio Attacks and Zero-Power Defenses,” Proceedings of the 2008 IEEE Symposium on Security and Privacy, Oakland, CA, May 18–21, 2008; and Jay Radcliffe, “Fact and Fiction: Defending Medical Device,” Black Hat 2013, July 31, 2013; It was not until June 2013 that the Federal Drug Administration (FDA) recommended vendors take voluntary steps “to prevent unauthorized access or modification to their medical devices.” See, FDA, “Cybersecurity for Medical Devices and Hospital Networks: FDA Safety Communication,” June 13, 2013.

55. Lillian Ablon, Martin Libicki, and Andrea Golay, Markets for Cybercrime Tools and Stolen Data, RAND Corporation, March 2014, pp. 13–14.

56. Intercrawler, “The Teenager Is the Author of BlackPOS/Kaptoxa Malware (Target), Several Other Breaches May Be Revealed Soon,” January 17, 2014; Jeremy Kirk, “Two Coders Closely Tied to Target-Related Malware,” computerworld.com, January 20, 2014; and Danny Yadron, Paul Ziobro, and Devlin Barrett, “Target Warned of Vulnerabilities Before Data Breach,” Wall Street Journal, February 14, 2014.

57. It is commonly asserted that 80 percent of known cyber attacks can be prevented by adopting five best practices: inventory authorized and unauthorized devices, inventory authorized and unauthorized software, develop and manage secure configurations for all devices, conduct continuous (automated) vulnerability assessment and remediation, and actively manage and control the use of administrative privileges. See, Center for Internet Security, “Cyber Hygiene Campaign,” accessed March 17, 2015 [www.cisecurity.org/about/CyberCampaign2014.cfm].

58. For more information, Pen Test Magazine has been a useful guide to emerging trends in the field since its founding in April 2011, as well as security conference presentations by hackers, which can often be found on YouTube soon after they are given.

59. Interviews with corporate and government cyber-security officials, 2012–2014; See also, James Kupsch and Barton Miller, “Manual vs. Automated Vulnerability Assessment: A Case Study,” Proceedings of the First International Workshop on Managing Insider Security Threats (MIST) West, West Lafayette, IN, June 15–19, 2009; and Matthew Finifter and David Wagner, “Exploring the Relationship Between Web Application Development Tools and Security,” Proceedings of the second USENIX Conference on Web Application Development, Portland, OR, June 15–16, 2011.

60. Women represent 11 percent of information security professionals. See, International Standard for Information Security (ISC) 2, Agents of Change: Women in the Information Security Profession, in partnership with Symantec, 2013; Catherine Pearce of the mobile security firm Neohapsis describes the community as “both liberal in its thinking, and sexist in its behavior. Conferences are also frankly dangerous. If you are a woman attending a conference, you have to be willing to punch someone in the face in public. Not all women want to do that.” Interview with Catherine Pearce, June 3, 2014.

61. Interview with a cyber-security professional, July 7, 2014.

62. The International Council of E-commerce Consultants contends that the training it provides to security researchers “is the world’s most advanced ethical hacking course with 19 of the most current security domains any ethical hacker will ever want to know when they are planning to beef up the information security posture of their organization. . . . You walk out the door with hacking skills that are highly in demand, as well as the internationally recognized certified ethical hacker certification!”; International Council of E-Commerce Consultants, “Ethical Hacking and Countermeasures to Become a Certified Ethical Hacker,” accessed May 4, 2015 [www.eccouncil.org/Certification/certified-ethical-hacker].

63. The purported hacker of the website called themselves “Eugene Bedford,” which was the character of a reformed hacker in the 1995 movie, Hackers. See, Megan Geuss, “Security Certification Group EC-Council’s Website Defaced with Snowden Passport,” ArsTechnica, February 23, 2014.

64. “Hacking Conferences,” Lanyrd, accessed March 17, 2015 [www.lanyrd.com/topics/hacking/]; and “Cybersecurity Conferences,” Lanyrd, accessed March 17, 2015 [lanyrd.com/topics/cyber-security/].

65. Black Hat, “USA 2009 Prospectus,” 2009; Paul Asadoorian, “Top 10 Things I Learned at Defcon 17,” Security Weekly, August 4, 2009; and Richard Reilly, “Black Hat and Defcon See Record Attendance—Even Without the Government Spooks,” VentureBeat, August 12, 2014.

66. Leyla Bilge and Tudor Dumitras, “Before We Knew It: An Empirical Study of Zero-Day Attacks in the Real World,” Proceedings of the 2012 ACM conference on Computer and Communications Security, Raleigh, NC, October 16–18, 2012.

67. Stefan Frei, “The Known Unknowns: Empirical Analysis of Publicly Unknown Security Vulnerabilities,” NSS Labs, December 2013; Barton Gellman and Ellen Nakashima, “U.S. Spy Agencies Mounted 231 Offensive Cyber-Operations in 2011, Documents Show,” Washington Post, August 20, 2013; and Ablon, Libicki, and Golay, Markets for Cybercrime Tools and Stolen Data.

68. For an entertaining immersion into the world of DEF CON, see, “DEFCON: The Documentary (2013),” YouTube, accessed March 17, 2015 [www.youtube.com/watch?v=rVwaIe6CiHw].

69. Interview with Jeff Moss, September 24, 2013.

70. US Commodity Futures Trading Commission, “CTFC Staff Advisory No. 14–21: Division of Swap Dealer and Intermediary Oversight,” February 26, 2014, accessed March 17, 2015 [www.cftc.gov/ucm/groups/public/@lrlettergeneral/documents/letter/14–21.pdf].

71. U.S. Code of Federal Regulations 45, “Public Welfare,” section 164.308, “Administrative Safeguards,” 2009; and Matthew Scholl et al., “An Introductory Resource Guide for Implementing the Health Insurance Portability and Accountability Act (HIPAA) Security Rule,” National Institute of Standards and Technology, US Department of Commerce, October 2008.

72. The security procedures listed in the PCI standards are not required by federal law, and as of the summer of 2014 had been mandated in only three states: Minnesota, Nevada, and Washington.

73. Javier Panzar and Paresh Dave, “Spending on Cyberattack Insurance Soars as Hacks Become More Common,” Los Angeles Times, February 10, 2015, p. C1.

74. Goldman Sachs also makes it a point to hire only smaller, boutique white-hat firms with highly specialized hacking skills, and to rotate security assessments among them so the same firm does not evaluate the same system repeatedly. Interview with Phil Venables, July 25, 2014.

75. Interviews with white-hat penetration testing firms, 2012–2014.

76. For example, a 2013 survey of 154 financial institutions in New York State found that while 85 percent used external white hats to conduct their penetration tests, only 13 percent commissioned them more than once a year—the minimum mandated by government regulations. See, New York State Department of Financial Services, Report on Cyber Security in the Banking Sector, May 2014, p. 5.

77. David Kennedy, keynote address at RVASEC, Richmond, Virginia, June 5, 2014. David Kennedy also rued, “I have folders of really cool and really sophisticated offensive tools, but I never get to use them during a pen test because I keep getting in the same way over and over for the past ten years.”

78. Interview with Brendan Conlon, April 15, 2014.

79. For representative examples of more advanced breaches, see, Rob Havelt and Wendel Guglielmetti, “Earth vs. The Giant Spider: Amazingly True Stories of Real Penetration Tests,” presentation at DEF CON 19, August 4–7, 2011; Deviant Ollam and Howard Payne, “Elevator Hacking: From the Pit to the Penthouse,” presentation at DEF CON 22, August 7–10, 2014; or, see many other presentations given at Black Hat or DEF CON, most of which are freely available on YouTube.

80. Interview with Nicholas Percoco, July 28, 2014.

81. At times, executives at the targeted institution will ask the white-hat firm to sanitize its report in order, for example, to remove mentions of critical vulnerabilities that were uncovered in soon-to-be-released software systems.

82. Interview with Ira Winkler, July 23, 2014.

83. Bob Stasio has found that many industries would rather spend money on firewalls and intrusion-detection systems from well-known (and expensive) cyber-security firms—like FireEye and Symantec—than less expensive, but often more effective systems from less prominent firms. Interview with Bob Stasio, June 30, 2014; In April 2014, FireEye refused to participate in an NSS Labs test of breach-detection systems—one of the most widely trusted resources in the information-security field—citing their methodology as “flawed.” FireEye claimed that NSS Labs’ finding of 147 “missed” samples the year before meant that “nobody could take this approach seriously.” See, Manish Gupta, “Real World vs Lab Testing: The FireEye Response to NSS Labs Breach Detection Systems Report,” FireEye, April 2, 2014.

84. Interview with Dan Guido, July 7, 2014.

85. Nico Golde, Kevin Redon, and Ravishankar Borgaonkar, “Weaponizing Femtocells: The Effect of Rogue Devices on Mobile Telecommunication,” Security in Telecommunications, Technische Universität Berlin, undated.

86. Between 2010 and 2013, Google paid out an average of $1,157 for each security vulnerability that was brought to its attention for its Chrome browser, while Mozilla paid an average of $3,000 for its Firefox. See, Matthew Finifter, Devdatta Akhawe, and David Wagner, “An Empirical Study of Vulnerability Rewards Programs,” paper presented at the USENIX Security Symposium, Washington, DC, August 14–16, 2013.

87. Interview with Nicholas Percoco, July 28, 2014. Percoco is a cofounder of the “I am the Cavalry” grassroots movement that is attempting to promote a more positive image of hacking, including its often under-appreciated and under-reported work on behalf of public safety and customer privacy.

88. Jared Allar, “Vulnerability Note VU#458007: Verizon Wireless Network Extender Multiple Vulnerabilities,” CERT Vulnerability Notes Database, July 15, 2013.

89. Jim Finkle, “Researchers Hack Verizon Device, Turn It into Mobile Spy Station,” Reuters, July 15, 2013. The iSEC Partners team had media-training sessions to practice and perfect the demonstrations and to keep their message simple; See also, Laura Sydell, “How Hackers Tapped into my Cellphone for Less Than $300,” National Public Radio, July 15, 2013; and Erica Fink and Laurie Segall, “Femtocell Hack Reveals Mobile Phones’ Calls, Texts and Photos,” CNN Money, July 15, 2013.

90. The title of both their presentations, which are available on YouTube, was: “I Can Hear You Now: Traffic Interception” and “Remote Mobile Phone Cloning with a Compromised CDMA Femtocell.”

91. For a comparable, publicly disclosed hack, see, Tobias Engel, “SS7: Locate, Track, Manipulate,” presentation at the 31st Chaos Communication Congress of the Chaos Computer Club, Hamburg, Germany, December 28, 2014. Given the growth of hackers testing software, hardware, and operating systems, there have been other examples of multiple teams independently uncovering the same vulnerability.

92. The name of the government agency is not revealed here because the conversation scheduled with the senior official was on background. Furthermore, although the security at this agency happened to be particularly poor during my visit, it may not have been representative of its overall security posture, and the basic security flaws could likely have been replicated at similar facilities.

93. Gavin Watson, Andrew Mason, and Richard Ackroyd, Social Engineering Penetration Testing: Executing Social Engineering Pen Tests, Assessments and Defense (Waltham, MA: Syngress Publications, 2014).

94. E-mail correspondence with Dalton Fury, May 19, 2014. Though Fury provides an illustrative example of attempting to break in by subverting the expected characteristics of an enemy, it is worth noting that the field of physical penetration testing is even more male-dominated then the white-hat-hacking community.

95. Ibid.; and Tina Dupuy, “He Hunted Osama Bin Laden, He Breaks into Nuclear Power Plants,” Atlantic Online, April 16, 2014.

96. Health Facilities Management and the American Society for Healthcare Engineering, “2012 Health Security Survey,” June 2012; and Lee Ann Jarousse and Suzanna Hoppszallern, “2013 Hospital Vendor & Visitor Access Control Survey,” Health Facilities Management and Hospitals & Health Networks, November 2013.

97. US Office of Personnel Management, “2014 Federal Employee Viewpoint Survey Results: Employees Influencing Change,” 2014, p. 41.

98. Curt Anderson, “Feds Break Up Major Florida-based Drug Theft Ring,” Associated Press, May 3, 2012; Although Eli Lilly and Company later filed a lawsuit against Tyco Integrated Security claiming that Tyco had failed to safeguard the confidential findings of the vulnerability assessment, Tyco denied the allegations, claiming that there is no proof. See, Kelly Knaub, “Tyco Can’t Ditch Suit over $60M Eli Lilly Warehouse Heist,” Law360, March 4, 2014 [www.law360.com/articles/515169/tyco-can-t-ditch-suit-over-60m-eli-lilly-warehouse-heist].

99. Amy Pavuk, “Drug Thief Linked to Orlando Heist,” Orlando Sentinel, August 9, 2013, p. A1. Eli Lilly and Co. later sued Tyco Integrated Security claiming that the thief must have gained access to the report.

100. Katie Dvorak, “33,000 Patient Records Stolen from California Radiology Facility,” CBS5 KPIX, June 12, 2014.

101. Abby Sewell, “L.A. County Finds 3,500 More Patients Affected by Data Breach,” Los Angeles Times, May 22, 2014, accessed March 17, 2015 [www.latimes.com/local/lanow/la-me-ln-county-data-breach-20140522-story.html].

102. Danielle Walker, “AvMed Breach Settlement Awards Plaintiffs Regardless of Suffered Fraud,” SC Magazine, March 2014, accessed March 17, 2015 [www.scmagazine.com/avmed-breach-settlement-awards-plaintiffs-regardless-of-suffered-fraud/article/340140/].

103. Chris Boyette, “New Jersey Teen Sneaks to Top of 1 World Trade Center, Police Say,” CNN, March 21, 2014.

104. Andrea Peyser, “WTC Wakeup Call for This Guy,” New York Post, April 4, 2014, p. 11.

105. In addition, the new requirements explicitly state the need for correcting vulnerabilities and conducting repeat penetration testing to verify those corrections.

106. Pete Herzog, OSSTMM 3: The Open Source Security Testing Methodology Manual, Institute for Security and Open Methodologies, 2010, p. 1.

107. One example was the theft of $2.1 million from a Barclays bank by eight criminals, one of whom was an insider and posed as an IT engineer to attach a keyboard/video/mouse, which costs about twenty dollars, to a computer in a London branch so they could transfer money remotely. See, Haroon Siddique, “£1.3m Barclays Heist—Eight Held,” The Guardian, September 21, 2013.

108. Verizon, 2011 Data Breach Investigations Report, April 2011, p. 40; and Verizon, 2014 Data Breach Investigations Report, April 2014, pp. 27–28.

109. Interview with Nicholas Percoco, July 28, 2014; and Interview with Charles Henderson, March 12, 2014.

110. TruTV, the episode first aired on December 25, 2007.

111. Interview with Chris Nickerson, June 12, 2014.

112. The book will be published by Elsevier B.V. and will be titled Red Team Testing: Offensive Security Techniques for Network Defense.

113. Chris Nickerson, “Hackers Are Like Curious Babies,” presentation at TEDxFullertonStreet, June 10, 2014.

114. Interview with Chris Nickerson, June 12, 2014.

115. One of the most interesting subfields in the hacking community is “locksport”—the recreational or competitive hobby of lock picking. Unlike criminal lock picking, locksport promotes transparency and full disclosure of how mechanical, electronic, and biometric locks can be bypassed. It is truly remarkable, when experienced firsthand, how relatively easy it is to defeat almost every lock that any thoughtful and diligent adversary would encounter in a supposedly secure facility. Search for “lock picking” videos on YouTube, especially those by the charming and obsessive Schuyler Towne, to learn how to pick locks.

116. Interview with Chris Nickerson, June 12, 2014.

117. Ibid.

118. Interview with Jayson Street, July 25, 2014.

119. Interview with Jayson Street, September 23, 2013.

120. Jayson Street, “Steal Everything, Kill Everyone, Cause Total Financial Ruin!” presentation at DEF CON 19, August 4–7 2011.

121. Interview with Jayson Street, September 23, 2013.

122. Interviews with Jayson Street, September 23, 2013 and July 25, 2014.

123. Steve Ragan, “Social Engineering: The Dangers of Positive Thinking,” CSOonline.com, January 5, 2015.

124. Interviews with Jayson Street, September 23, 2013 and July 25, 2014.

125. Jayson Street, “How to Channel Your Inner Henry Rollins,” presentation at DEF CON 20, July 26–29, 2012.

126. A 2014 survey of 1,600 IT security professionals found that while “more than ninety-six percent of organizations experienced a significant IT security incident in the past year . . . only thirty-three percent have confidence that their organizations would improve those security measures.” See, Forescout, IDG Survey: State of IT Cyber Defense Maturity, July 2014.

127. Interview with Dino Dai Zovi, July 18, 2014.

128. Interview with Jayson Street, July 25, 2014.

CHAPTER SIX

1. Supreme Court of Tennessee, The State of Tennessee v. John Thomas Scopes, 1925.

2. World Health Assembly, “Global Eradication of Poliomyelitis by the Year 2000,” WHA41.28, May 13, 1988.

3. Global Polio Eradication Initiative, Budgetary Implications of the GPEI Strategic Plan and Financial Resource Requirements 2009–2013, January 2009, p. 5; and “End Polio Now,” Rotary International, accessed March 17, 2015 [www.endpolio.org/about-polio].

4. World Health Organization, “Poliomyelitis: Fact Sheet N144,” April 2013; Global Polio Eradication Initiative, Global Polio Eradication Progress 2000 (Geneva, Switzerland: World Health Organization, 2001); and Centers for Disease Control and Prevention, “CDC’s Work to Eradicate Polio,” updated September 2014.

5. Centers for Disease Control and Prevention, “Progress Toward Interruption of Wild Poliovirus Transmission–Worldwide, 2009,” March 14, 2010.

6. Gregory Pirio and Judith Kaufmann, “Polio Eradication Is Just over the Horizon: The Challenges of Global Resource Mobilization,” Journal of Health Communication: International Perspectives 15, supplement 1, 2010, pp. 66–83.

7. Interview with Gregory Pirio, July 18, 2013.

8. Interviews with Ellyn Ogden, April 25, 2012 and July 10, 2013.

9. Global Polio Eradication Initiative, Polio Eradication and Endgame Strategic Plan 2013–2018, 2013, p. 97.

10. Independent Monitoring Board of the Global Polio Eradication Initiative, Eleventh Report, May 2015, pp. 7, 10.

11. Barry Staw, “Is Group Creativity Really an Oxymoron? Some Thoughts on Bridging the Cohesion-Creativity Divide,” in Elizabeth Mannix, Margaret Neal, and Jack Goncalo, eds. Creativity in Groups, Research on Managing Groups and Teams, vol. 12 (Bradford, UK: Emerald Publishing, 2009), pp. 311–323.

12. The website also features Mateski’s invaluable Laws of Red Teaming, of which there are fifty. See, “The Laws of Red Teaming,” Red Team Journal, accessed August 27, 2015 [www.redteamjournal.com/red-teaming-laws/]. He leads with Red Teaming Law #1: “The more powerful the stakeholders, the more at stake, the less interest in red teaming. This law trumps all other laws.”

13. Interview with Mark Mateski, April 18, 2014.

14. Interviews with Mark Mateski, April 18, 2014 and July 25, 2014.

15. Interview with Chris Nickerson, June 12, 2014.

16. World War Z, directed by Marc Forster (Paramount Pictures, 2013).

17. Babylonian Talmud, “Tractate Sanhedrin: Come and Hear,” Folio 17a. Princeton University professor Michael Walzer interprets this passage: “The absence of dissent means that there wasn’t an adequate deliberation.” See, Michael Walzer, “Is the Right Choice a Good Bargain?” New York Review of Books, 62(4), March 5, 2015.

18. Robert Kennedy, Thirteen Days: A Memoir of the Cuban Missile Crisis (New York: W.W. Norton & Company, 1969), p. 86.

19. In experimental settings, authentic dissenters stimulate more creative solutions than an individual assigned to a devil’s advocate role. See, Charlan Nemeth, Keith Brown, and John Rogers, “Devil’s Advocate Versus Authentic Dissent: Stimulating Quantity and Quality,” European Journal of Social Psychology, 31, 2001, pp. 707–720.

20. Nicholas Hilling, Procedure at the Roman Curia (New York: Wagner, 1909), pp. 41–42.

21. The Pentagon Papers, Gravel Edition, vol. 4 (Boston, MA: Beacon Press, 1971), pp. 615–619.

22. George Ball, The Past Has Another Pattern (New York: W.W. Norton & Company, 1982), p. 384.

23. George Reedy, The Twilight of the Presidency (Cleveland, OH: World Publishing Company, 1970), p. 11.

24. James Thomson, “How Could Vietnam Happen? An Autopsy,” Atlantic Monthly, 221(4), April 1968, pp. 47–53.

25. John Schlight, The War in South Vietnam: The Years of the Offensive, 1965–1968 (Washington, DC: Department of the US Air Force, 1989).

26. Stefan Schulz-Hardt, Marc Jochims, and Dieter Frey, “Productive Conflict in Group Decision Making: Genuine and Contrived Dissent as Strategies to Counteract Biased Information Seeking,” Organizational Behavior and Human Decision Processes, 88, 2002, pp. 563–586.

27. Michael Gordon, “The Iraq Red Team,” Foreign Policy, September 24, 2012; Editorial Board, “The U.S. Is Not Ready for a Cyberwar,” Washington Post, March 11, 2013, p. A14; Freedom of Information Act Request made by Ralph Hutchison to the US Department of Energy, Oak Ridge Environmental Peace Alliance, April 24, 2014, accessed March 17, 2015 [www.orepa.org/wp-content/uploads/2014/04/Red-Team-FOIA.pdf]; and Bill Gertz, “Military Report: Terms ‘Jihad,’ ‘Islamist’ Needed,” Washington Times, October 20, 2008, p. A1.

28. Mark Perry, “Red Team: Centcom Thinks Outside the Box on Hamas and Hezbollah,” Foreign Policy, June 30, 2010.

29. Bilal Saab, “What Do Red Teams Really Do?” Foreign Policy, September 3, 2010.

30. Interview with Gen. David Petraeus, February 19, 2014; and Interview with an Army colonel, January 2011.

31. Michael Gordon, “The Iraq Red Team.” For more see, Michael Gordon and Gen. Bernard Trainor, The Endgame: The Inside Story of the Struggle for Iraq, From George W. Bush to Barack Obama (New York: Pantheon Books, 2012), pp. 95–97.

32. George Casey, “About that Red Team Report,” Foreign Policy, September 27, 2012.

33. Interview with a former PACOM intelligence official, May 2014.

34. Lindsay Toler, “KSDK Investigation on School Safety in Kirkwood Reveals Journalists Are the Worst,” St. Louis Riverfront Times, January 17, 2014, accessed March 17, 2015 [www.blogs.riverfronttimes.com/dailyrft/2014/01/ksdk_kirkwood_lockdown.php].

35. Jessica Bock, “KSDK Reporter Working on School Safety Story Prompted Kirkwood High Lockdown,” St. Louis Post-Dispatch, January 17, 2014, p. A1.

36. KSDK, “News Channel 5 Report on School Safety,” January 16, 2014, accessed March 17, 2015 [www.ksdk.com/story/news/local/2014/01/16/newschannel-5-statement-school-safety/4531859/].

37. Ibid.

38. NBC, “Rossen Reports: New Device Can Open Hotel Room Locks,” Today Show, December 6, 2012; and Onity United Technologies, “Information for Onity HT and ADVANCE Customers,” August 2012.

39. Interview with a Marine Corps colonel, May 2013; and Interview with an ISAF staff officer, November 2013.

40. Hearing of the House Foreign Affairs Committee, “U.S. Strategy in Afghanistan,” December 2, 2009.

41. Bill Roggio and Lisa Lundquist, “Green-on-Blue Attacks in Afghanistan: The Data,” The Long War Journal, August 23, 2012, data updated April 8, 2015.

42. Interview with an ISAF staff officer, November 2013.

43. M. G. Siegler, “The VP of Devil’s Advocacy,” TechCrunch, July 27, 2014.

44. David Fahrenthold, “Unrequired Reading,” Washington Post, May 3, 2014, p. A1. On November 12, 2014, the House of Representatives unanimously voted in favor of the Government Reports Elimination Act (H.R. 4194), which would eliminate 321 reports from twenty-nine federal agencies.

45. US House of Representatives, National Defense Authorization Act for Fiscal Year 2003 Conference Report, November 12, 2002.

46. US Senate, Intelligence Reform and Terrorism Prevention Act of 2004 Conference Report, December 8, 2004.

47. US Senate, S. 2845, National Intelligence Reform Act of 2004, October 6, 2004.

48. P.L. 108–458, Intelligence Reform and Terrorism Prevention Act of 2004, December 17, 2004.

49. The SAFE Port Act (H.R. 4954) was passed into law on March 14, 2006. Seven others died in the Senate or House: the Department of Homeland Security Authorization Act for Fiscal Year 2006 (H.R. 1817), John Warner National Defense Authorization Act for Fiscal Year 2007 (S. 2766), Chemical Facility Anti-Terrorism Act of 2006 (H.R. 5695), Rail and Public Transportation Security Act of 2006 (H.R. 5714), Department of Homeland Security Authorization Act for Fiscal Year 2007 (H.R. 5814), Department of Homeland Security Authorization Act for Fiscal Year 2008 (H.R. 1684), and Chemical Facility Anti-Terrorism Act of 2008 (H.R. 5577).

50. Office of Senator Angus King, “Senate Intelligence Committee Approves King and Rubio Amendment to Provide Independent Check on Targeting Decisions,” November 6, 2013.

51. P.L. 113–126, Intelligence Authorization Act for Fiscal Year 2014, July 7, 2014. Reportedly, the language contained in the final version of the Act was similar to what King and Rubio had originally proposed. See, Office of Senator Marco Rubio, “Senate Intelligence Committee Approves Rubio & King Amendment to Provide Independent Check on Targeting Decisions,” November 6, 2013. A competing bill would have declassified the alternative-analysis report after ten years, but that provision was removed from the legislation that became law.

52. Marco Rubio, “Senate Intelligence Committee Approves Rubio & King Amendment to Provide Independent Check Targeting Decision.”

53. Interviews with Senate and House Intelligence Committee staffers, 2013 and 2014; moreover, seven of the eight US citizens believed to have been killed by US drone strikes were not knowingly targeted, so they would not have benefited from any additional review. See, Micah Zenko, “The United States Does Not Know Who It’s Killing,” Foreign Policy, April 23, 2015 [www.foreignpolicy.com/2015/04/23/the-united-states-does-not-know-who-its-killing-drone-strike-deaths-pakistan/].

54. For the proposal to permanently establish an independent strategic advisory board within the National Security Council (NSC), see David Gompert, Hans Binnendijk, and Bonny Lin, Blinders, Blunders, and Wars: What America and China Can Learn, RAND Corporation, 2014, pp. 203–208. This red team concept is intriguing, though as the authors acknowledge, a permanent board would most likely become institutionally captured by the NSC.

55. Defense Science Board Task Force on the Role and Status of DoD Red Teaming Activities, p. 1.

56. Susan Straus et al., Innovative Leader Development: Evaluation of the U.S. Asymmetric Warfare Adaptive Leader Program, RAND Corporation, 2014; and interview with a retired military officer, May 2015. One positive step toward improving the impact of red-teaming instruction was an effort by retired military officers in 2015 to draft a joint doctrine note specifically for red teaming. Joint doctrine notes provide non-authoritative, common fundamental guidance for how the armed services should develop and employ military concepts.

57. William Perry and John Abizaid, Ensuring a Strong U.S. Defense for the Future: The National Defense Panel Review of the 2014 Quadrennial Defense Review, United States Institute of Peace, July 31, 2014, p. 65.

58. All of the NDP members were defense industry lobbyists or corporate board members previously, at that time, or soon thereafter.

59. See, Perry and Abizaid, Ensuring a Strong U.S. Defense for the Future, appendix 6, pp. 69–72.

60. The suggestion comes from Jim Thomas, vice president and director of studies at the Center for Strategic and Budgetary Assessments, suggested during a February 26, 2013 hearing, “The Quadrennial Defense Review: Process, Policy, and Perspectives,” of the House Armed Services Committee, Subcommittee on Oversight and Investigations.

61. Interview with Brig. Tom Longland, November 25, 2014.

62. A Bain & Company survey of management tools offers a warning for new tools, which could be applied to red teaming: “Hyperbole surrounding the trendiest of tools often leads to unrealistic expectations and disappointing results.” See, Darrell Rigby, Management Tools 2013: An Executive’s Guide, Bain & Company, 2013, p. 11.

63. Chris Thornton et al., “Automated Testing of Physical Security: Red Teaming Through Machine Learning,” Computational Intelligence, published online February 27, 2014; Hussein Abbass, “Computational Red Teaming: Past, Present and Future,” IEEE Computational Intelligence Magazine, 6(1), February 2011, pp. 30–42; and Philip Hingston, Mike Preuss, and Daniel Spierling, “RedTNet: A Network Model for Strategy Games,” Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2010, Barcelona, Spain, July 2010.

64. Eric Davisson and Ruben Alejandro, “Abuse of Blind Automation in Security Tools,” presentation at DEF CON 22, August 8, 2014.

65. Interview with Samuel Visner, December 1, 2014.

66. Raphael Mudge, “Cortana: Rise of the Automated Red Team,” presentation at DEF CON 20, August 28, 2012, accessed March 17, 2015 [www.youtube.com/watch?v=Eca1k-lgih4].

67. Philip Polstra, Hacking and Penetration Testing with Low Power Devices (Boston, MA: Syngress, 2012).

68. Gregg Schudel and Bardley Wood, “Adversary Work Factor as a Metric for Information Assurance,” Proceedings of the 2000 New Security Paradigm Workshop, 2000, pp. 23–30.

69. Interview with a senior intelligence community official, April 2014.

70. Silas Allen, “University of Oklahoma Researchers Develop Video Game to Test for Biases,” Oklahoman, October 14, 2013.

71. Interview with a senior intelligence community official, April 2014.

72. Interview with Nicholas Percoco, July 28, 2014.

73. Tom Head, eds., Conversations with Carl Sagan (Jackson, MS: University Press of Mississippi, 2006), p. 135.