The interconnecting databases developed for this project involved the review of hundreds of thousands of documents. The basic questions we sought to answer were which agencies do work at the top secret level within the U.S. government, what type of work it is, where this work physically takes place, and what private contractors are involved. Then we asked how much of this effort began after 9/11, and how much had efforts under way before 9/11 expanded since then.
The databases included:
We mined four basic sources of data to build these databases and ultimately used the following feeder sources (from the hundreds of thousands collected) for the Top Secret America website (http://projects.washingtonpost.com/top-secret-america/):
In total, 112,000 individual files totaling 520 GB of data were collected. The databases we built describing over 700 government entities and 1,900 companies included 640,000 fields. Over 10,000 locations were geocoded; at the Washington Post, web specialists, researchers, interns, and copyeditors helped with input, design, and fact-checking.
For additional information on the methodology used in the project, see: http://projects.washingtonpost.com/top-secret-america/articles/methodology/.
James Bamford. A Pretext for War: 9/11, Iraq, and the Abuse of America’s Intelligence Agencies (New York: Doubleday, 2004).
——. The Shadow Factory: The Ultra-Secret NSA from 9/11 to the Eavesdropping on America (New York: Doubleday, 2008).
Gary Berntsen and Ralph Pezzullo. Jawbreaker: The Attack on Bin Laden and Al-Qaeda: A Personal Account by the CIA’s Key Field Commander (New York: Crown, 2005).
George W. Bush. Decision Points (New York: Crown, 2010).
Richard Clarke. Against All Enemies: Inside America’s War on Terror (New York: Free Press, 2004).
Steve Coll. Ghost Wars: The Secret History of the CIA, Afghanistan and Bin Laden, from the Soviet Invasion to September 10, 2001 (New York: Penguin Press, 2004).
Bob Drogin. Curveball: Spies, Lies, and the Con Man Who Caused a War (New York: Random House, 2007).
General Tommy Franks (with Malcolm McConnell). American Soldier (New York: HarperCollins Publishers, 2004).
Roger Z. George and Harvey Rishikof. The National Security Enterprise (Washington, DC: Georgetown University Press, 2011).
Bradley Graham. By His Own Rules: The Ambitions, Successes, and Ultimate Failures of Donald Rumsfeld (New York: Public Affairs, 2009).
Rebecca Grant. The First 600 Days of Combat: The U.S. Air Force in the Global War on Terrorism (Washington, DC: IRIS Press, 2004).
Benjamin S. Lambeth. Air Power Against Terror: America’s Conduct of Operation Enduring Freedom (Santa Monica, CA: Rand Corporation, 2005).
Matt J. Martin (with Charles W. Sasser). Predator: The Remote-Control Air War over Iraq and Afghanistan: A Pilot’s Story (Minneapolis, MN: Zenith Press, 2010).
General Richard B. Myers, USAF, Ret. (with Malcolm McConnell). Eyes on the Horizon: Serving on the Front Lines of National Security (New York: Threshold Editions [A Division of Simon & Schuster, Inc.], 2009).
Sean Naylor. Not a Good Day to Die: The Untold Story of Operation Anaconda (New York: Berkley Publishing Group, 2005).
Bruce Reidel. Deadly Embrace (Washington, DC: Brooking Institution Press, 2011).
James Risen. State of War: The Secret History of the CIA and the Bush Administration (New York: Free Press, 2006).
Donald Rumsfeld. Known and Unknown: A Memoir (New York: Sentinel, 2011).
Tim Shorrock. Spies for Hire: The Secret World of Intelligence Outsourcing (New York: Simon & Schuster, 2008).
Gary C. Shroen. First In: An Insider’s Account of How the CIA Spearheaded the War on Terror in Afghanistan (New York: Ballantine Books/Presidio Press, 2005).
George Tenet (with Bill Harlow). At the Center of the Storm (New York: HarperCollins Publishers, 2007).
Paul Thompson. The Terror Timeline (New York: HarperCollins Publishers, 2004).
Bob Woodward. Bush at War (New York: Simon & Schuster, 2002).
———. Obama’s Wars (New York: Simon & Schuster, 2010).
———. Plan of Attack (New York: Simon & Schuster, 2004).
———. State of Denial: Bush at War, Part III (New York: Simon & Schuster, 2006).
We utilized countless budget books from various national security agencies; Congressional hearings and committee reports; and reports of the General Accountability Office, Congressional Research Service, and the Inspector General’s offices of the Defense Department, the military services, the Department of Homeland Security, and the Department of Justice. At the federal and state levels, we collected well over 1,000 warnings and intelligence reports from intelligence community members and state fusion centers. In addition:
The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks Upon the United States, Authorized Edition (New York: W. W. Norton and Company, 2003).
Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction
Office of the Director of National Intelligence, National Intelligence: A Consumer’s Guide, 2009.
Report on the U.S. Intelligence Community’s Prewar Intelligence Assessments on Iraq
U.S. Air Force (CENTAF), “Fast and Final: Operation Iraqi Freedom,” 22 March 2004, Unclassified Powerpoint Briefing.
U.S. Army Special Operations Command, Weapon of Choice: ARSOF in Afghanistan (Ft. Leavenworth, KS: Combat Studies Institute Press, 2003).
U.S. Army Special Operations Command, All Roads Lead to Baghdad: Army Special Operations Forces in Iraq (Ft. Bragg, NC: USASOC History Office, 2005).
U.S. Army, The United States Army in Afghanistan: Operation Enduring Freedom.
U.S. Special Operations Command (SOCOM), History of SOCOM, 6th edition, 31 March 2008.
We also used various published and unpublished papers of the National Defense University and the war colleges and specialized higher education institutions of the Defense Department and the intelligence community.
We also found invaluable the near-daily newsletter produced by Steve Aftergood of the Federation of American Scientists, called SecrecyNews.