NOTES

Introduction: How to Lie with Statistics

1. Umberto Eco, Serendipities: Language and Lunacy (London: Orion, 1998).

2. Robert Matthews, “Storks Deliver Babies (p = 0.008),” Teaching Statistics 22, no. 2 (June 2000), 36–38. Research papers in social science typically say that a relationship is “statistically significant” if p = 0.05, which means that if there was no relationship at all, a pattern at least as clear as the one observed would occur just one in twenty times. The stork paper boasted p = 0.008, which means that if there was in fact no relationship between storks and births, a pattern as clear as the one observed would occur just one in 125 times. The tradition of applying such a statistical significance test is regrettable, for reasons we shall not go into now.

3. Conrad Keating, Smoking Kills (Oxford, UK: Signal Books, 2009), xv.

4. See, among others, Peter Armitage, “Obituary: Sir Austin Bradford Hill, 1897–1991,” Journal of the Royal Statistical Society, Series A (Statistics in Society), 154, no. 3 (1991), 482–84, JSTOR, www.jstor.org/stable/2983156.

5. Keating, Smoking Kills, 85–90.

6. Keating, 113.

7. John P. A. Ioannidis, “A Fiasco in the Making?,” Stat, March 17, 2020, https://www.statnews.com/2020/03/17/a-fiasco-in-the-making-as-the-coronavirus-pandemic-takes-hold-we-are-making-decisions-without-reliable-data/.

8. “Taiwan Says WHO Failed to Act on Coronavirus Transmission Warning,” Financial Times, March 20, 2020, https://www.ft.com/content/2a70a02a-644a-11ea-a6cd-df28cc3c6a68.

9. Demetri Sevastopulo and Hannah Kuchler, “Donald Trump’s Chaotic Coronavirus Crisis,” Financial Times, March 27, 2020, https://www.ft.com/content/80aa0b58-7010-11ea-9bca-bf503995cd6f.

10. David Card, “Origins of the Unemployment Rate: The Lasting Legacy of Measurement without Theory,” UC Berkeley and National Bureau of Economic Research Working Paper, February 2011, http://davidcard.berkeley.edu/papers/origins-of-unemployment.pdf.

11. Naomi Oreskes and Eric Conway, Merchants of Doubt (New York: Bloomsbury, 2010), chap. 1; and Robert Proctor, Golden Holocaust (Berkeley: University of California Press, 2011).

 12. Unknown—No Title—August 21, 1969, Brown & Williamson Record, https://www.industrydocumentslibrary.ucsf.edu/tobacco/docs/xqkd0134.

13. Kari Edwards and Edward Smith, “A Disconfirmation Bias in the Evaluation of Arguments,” Journal of Personality and Social Psychology 71, no. 1 (1996), 5–24.

14. Oreskes and Conway, Merchants of Doubt.

15. Michael Lewis, “Has Anyone Seen the President?,” Bloomberg, February 9, 2018, https://www.bloomberg.com/opinion/articles/2018-02-09/has-anyone-seen-the-president.

16. Brendan Nyhan, “Why Fears of Fake News Are Overhyped,” Medium, February 4, 2019; and Gillian Tett, “The Kids Are Alright: The Truth about Fake News,” Financial Times, February 6, 2019, https://www.ft.com/content/d8f43574-29a1-11e9-a5ab-ff8ef2b976c7?desktop=true&segmentId=7c8f09b9-9b61-4fbb-9430-9208a9e233c8.

17. “Health Warning Required on Cigarette Packs,” Congressional Quarterly Almanac 21 (1965), 344–51, https://library.cqpress.com/cqalmanac/document.php?id=cqal65-1259268; and Alex Reinhart, “Huff and Puff,” Significance 11, no. 4 (October 3, 2014).

18. Andrew Gelman, “Statistics for Cigarette Sellers,” Chance 25, no. 3 (2012); Reinhart, “Huff and Puff.”

19. How to Lie with Smoking Statistics is stored in the Tobacco Industry Documents library. Alex Reinhart pieced together the manuscript and various documents pertaining to the project: Reinhart, “The History of ‘How to Lie with Smoking Statistics,’” October 4, 2014, https://www.refsmmat.com/articles/smoking-statistics.html.

20. Suzana Herculano-Houzel, “What Is So Special about the Human Brain?,” TED Talk, 2013: https://www.ted.com/talks/suzana_herculano_houzel_what_is_so_special_about_the_human_brain/transcript?ga_source=embed&ga_medium=embed&ga_campaign=embedT.

21. On Galileo’s telescope: “Refusing to Look,” The Renaissance Mathematicus (blog), August 23, 2012, https://thonyc.wordpress.com/2012/08/23/refusing-to-look/; and Ryan D. Tweney, “How the Telescope Changed Our Minds,” Wired, October 2, 2008, https://www.wired.com/2008/10/how-the-telesco/; and https://web.archive.org/web/20160829064012/https://thekindlyones.org/2010/10/13/refusing-to-look-through-galileos-telescope/.

Rule One: Search Your Feelings

1. Also known as Star Wars: Episode V; screenplay by Leigh Brackett and Lawrence Kasdan.

2. The van Meegeren case is described in John Godley, The Master Forger (London: Home and Van Thal, 1951); Van Meegeren: A Case Study (London: Nelson, 1967); Noah Charney, The Art of Forgery: The Minds, Motives and Methods of Master Forgers (London: Phaidon, 2015); Frank Wynne, I Was Vermeer (London: Bloomsbury, 2007); Edward Dolnick, The Forger's Spell (New York: Harper Perennial, 2009); the BBC TV program Fake or Fortune (series 1, program 3, 2011); a series of blog posts by Errol Morris titled “Bamboozling Ourselves,” starting on the New York Times website, May 20, 2009; the Boijmans Museum film Van Meegeren’s Fake Vermeers (2010, available on YouTube at https://www.youtube.com/watch?v=NnnkuOz08GQ); and particularly Jonathan Lopez, The Man Who Made Vermeers (London: Houghton Mifflin, 2009).

3. Different accounts exist of exactly how van Meegeren made this confession—another account has van Meegeren equating himself with the Dutch master more directly: “The painting in Göring’s hands is not, as you assume, a Vermeer of Delft, but a van Meegeren!” The quotation in the text is from Frank Wynne’s book I Was Vermeer.

4. Ziva Kunda, “Motivated Inference: Self-Serving Generation and Evaluation of Causal Theories,” Journal of Personality and Social Psychology 53, no. 4 (1987), 636–47.

5. Stephen Jay Gould, “The Median Isn’t the Message,” AMA Journal of Ethics (January 2013), DOI: http://dx.doi.org/10.1001/virtualmentor.2013.15.1.mnar1-1301.

6. This experiment was described on NPR’s Hidden Brain podcast: “You 2.0: The Ostrich Effect,” August 6, 2018, https://www.npr.org/templates/transcript/transcript.php?storyId=636133086.

7. Nachum Sicherman et al., “Financial Attention,” Review of Financial Studies 29, no. 4 (April 1, 2016), 863–97, https://doi.org/10.1093/rfs/hhv073.

8. “Viral Post about Someone’s Uncle’s Coronavirus Advice Is Not All It’s Cracked Up to Be,” Full Fact, March 5, 2020, https://fullfact.org/online/coronavirus-claims-symptoms-viral/.

9. Guy Mayraz, “Wishful Thinking,” October 25, 2011, http://dx.doi.org/10.2139/ssrn.1955644.

10. Linda Babcock and George Loewenstein, “Explaining Bargaining Impasse: The Role of Self-Serving Biases,” Journal of Economic Perspectives 11, no. 1 (1997), 109–26, https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.11.1.109.

11. A good summary is Dan Kahan’s blog post “What Is Motivated Reasoning? How Does It Work?,” reproduced at http://blogs.discovermagazine.com/intersection/2011/05/05/what-is-motivated-reasoning-how-does-it-work-dan-kahan-answers/#.WN5zJ_nyuUm. An excellent survey is Ziva Kunda, “The Case for Motivated Reasoning,” Psychological Bulletin 108, no. 3 (1990), 480–98, http://dx.doi.org/10.1037/0033-2909.108.3.480.

12. S. C. Kalichman, L. Eaton, and C. Cherry, “‘There Is No Proof That HIV Causes AIDS’: AIDS Denialism Beliefs among People Living with HIV/AIDS,” Journal of Behavioral Medicine 33, no. 6 (2010), 432–40, DOI: 10.1007/s10865-010-9275-7; and A. B. Hutchinson et al., “Conspiracy Beliefs and Trust in Information about HIV/AIDS among Minority Men Who Have Sex with Men,” Journal of Acquired Immune Deficiency Syndrome 45, no. 5 (August 15, 2007), 603–5.

13. Tim Harford, “Why It’s Too Tempting to Believe the Oxford Study,” Financial Times, March 27, 2020, https://www.ft.com/content/14df8908-6f47-11ea-9bca-bf503995cd6f.

14. Keith E. Stanovich, Richard F. West, and Maggie E. Toplak, “Myside Bias, Rational Thinking, and Intelligence,” Current Directions in Psychological Science 22, no. 4 (August 2013), 259–64, DOI: 10.1177/0963721413480174.

15. Charles S. Taber and Milton Lodge, “Motivated Skepticism in the Evaluation of Political Beliefs,” American Journal of Political Science 50, no. 3 (July 2006), 755–69, JSTOR, www.jstor.org/stable/3694247.

16. Kevin Quealy, “The More Education Republicans Have, the Less They Tend to Believe in Climate Change,” New York Times, November 14, 2017, https://www.nytimes.com/interactive/2017/11/14/upshot/climate-change-by-education.html.

17. Caitlin Drummond and Baruch Fischhoff, “Individuals with Greater Science Literacy and Education Have More Polarized Beliefs on Controversial Science Topics,” Proceedings of the National Academy of Sciences of the United States of America, August 21, 2017, http://www.pnas.org/content/early/2017/08/15/1704882114.

 18. Charles Lord, L. Ross, and M. R. Lepper, “Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence,” Journal of Personality and Social Psychology 37, no. 11 (1979), 2098–2109.

19. Nicholas Epley and Thomas Gilovich, “The Mechanics of Motivated Reasoning,” Journal of Economic Perspectives 30, no. 3 (2016), 133–40, https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.30.3.133.

20. Ari LeVaux, “Climate Change Threatens Montana’s Barley Farmers—and Possibly Your Beer,” Food and Environment Research Network, December 13, 2017, https://thefern.org/2017/12/climate-change-threatens-montanas-barley-farmers-possibly-beer/.

21. Author correspondence with Kris De Meyer, October 27, 2018.

22. Gordon Pennycook et al., “Understanding and Reducing the Spread of Misinformation Online,” PsyArXiv, November 13, 2019, DOI: 10.31234/osf.io/3n9u8; also see Oliver Burkeman, “How to Stop the Spread of Fake News? Pause for a Moment,” Guardian, February 7, 2020, https://www.theguardian.com/lifeandstyle/2020/feb/07/how-to-stop-spread-of-fake-news-oliver-burkeman.

23. G. Pennycook and D. G. Rand, “Lazy, Not Biased: Susceptibility to Partisan Fake News Is Better Explained by Lack of Reasoning Than by Motivated Reasoning,” Cognition, 2018, https://doi.org/10.1016/j.cognition.2018.06.011.

24. Shane Frederick, “Cognitive Reflection and Decision Making,” Journal of Economic Perspectives 19, no. 4 (2005), 25–42, DOI: 10.1257/089533005775196732.

25. Diane Wolf, Beyond Anne Frank: Hidden Children and Postwar Families in Holland (Berkeley: University of California Press, 2007), Table 1, citing Raul Hilberg, The Destruction of the European Jews (1985).

Rule Two: Ponder Your Personal Experience

1. Muhammad Yunus, in “Stephen Covey Interviews Grameen Bank Founder Muhammad Yunus,” SocialBusinessPedia, last updated November 21, 2016, http://socialbusinesspedia.com/wiki/details/248.

2. Transport for London, Travel in London: Report 11 (2018), figure 10.8, 202, http://content.tfl.gov.uk/travel-in-london-report-11.pdf.

3. These numbers were revealed in a Freedom of Information request, March 28, 2011 (https://www.whatdotheyknow.com/request/journey_demand_and_service_suppl), and they are nicely summarized at the website Ian Visits, “London Tube Train Capacities,” August 5, 2106, https://www.ianvisits.co.uk/blog/2016/08/05/london-tube-train-capacities/.

4. Transport for London, Travel in London: Report 4 (2011), 5, http://content.tfl.gov.uk/travel-in-london-report-4.pdf.

5. Author interview with Lauren Sager Weinstein and Dale Campbell of TfL, July 9, 2019.

6. Ipsos Mori, Perils of Perception 2017, December 5, 2017, https://www.ipsos.com/ipsos-mori/en-uk/perils-perception-2017.

7. “‘No Link between MMR and Autism,’ Major Study Finds,” NHS News, March 5, 2019, https://www.nhs.uk/news/medication/no-link-between-mmr-and-autism-major-study-finds/.

8. National Institute of Child Health and Clinical Development, “When Do Children Usually Show Symptoms of Autism?,” January 31, 2017, https://www.nichd.nih.gov/health/topics/autism/conditioninfo/symptoms-appear.

 9. David McRaney, interview with Lee Ross, “Naive Realism,” episode 62 of You Are Not So Smart, November 9, 2015, https://youarenotsosmart.com/2015/11/09/yanss-062-why-you-often-believe-people-who-see-the-world-differently-are-wrong/; and Tom Gilovich and Lee Ross, The Wisest One in the Room (New York: Free Press, 2016).

10. Ipsos Mori, Perils of Perception.

11. David Dranove et al., “Is More Information Better? The Effects of ‘Report Cards’ on Health Care Providers,” National Bureau of Economic Research Working Paper 8697 (2002), http://www.nber.org/papers/w8697.

12. Charles Goodhart, “Problems of Monetary Management: The U.K. Experience,” in Inflation, Depression, and Economic Policy in the West, ed. Anthony S. Courakis (Lanham, MD: Rowman & Littlefield, 1981), 111–46. The original paper was presented at a conference in 1975.

13. Donald T. Campbell, “Assessing the Impact of Planned Social Change,” Evaluation and Program Planning 2, no. 1 (1979). An earlier version was published in 1976 and a conference paper existed in 1974.

14. Abhijit Banerjee, Dean S. Karlan, and Jonathan Zinman, “Six Randomized Evaluations of Microcredit: Introduction and Further Steps,” American Economic Journal: Applied Economics 7, no. 1 (2015), 1–21; and Rachel Meager, “Understanding the Average Effect of Microcredit,” VoxDev, July 1, 2019, https://voxdev.org/topic/methods-measurement/understanding-average-effect-microcredit.

15. Anna Rosling Rönnlund, “See How the Rest of the World Lives, Organized by Income,” a talk presented at a TED conference, April 2017, https://www.ted.com/talks/anna_rosling_ronnlund_see_how_the_rest_of_the_world_lives_organized_by_income?language=en.

Rule Three: Avoid Premature Enumeration

1. Dr. Lucy Smith was interviewed by me and my colleague Richard Fenton-Smith for an episode of More or Less broadcast on BBC Radio 4, June 8, 2018, https://www.bbc.co.uk/programmes/p069jd0p. The account here is based on our broadcast interview, discussions over email, and a phone interview I conducted with Dr. Smith on August 12, 2019. Dr. Smith’s interviews with people who had lost a baby at between twenty and twenty-four weeks of pregnancy are at “Losing a Baby at 20-24 Weeks of Pregnancy,” healthtalk.org.

2. Merian F. MacDorman et al., “International Comparisons of Infant Mortality and Related Factors: United States and Europe, 2010,” National Vital Statistics Reports, September 24, 2014.

3. Denis Campbell, “Concern at Rising Infant Mortality Rate in England and Wales,” Guardian, March 15, 2018, https://www.theguardian.com/society/2018/mar/15/concern-at-rising-infant-mortality-rate-in-england-and-wales.

4. Peter Davis et al., “Rising Infant Mortality Rates in England and Wales—We Need to Understand Gestation Specific Mortality,” BMJ (May 8, 2018), 361.

5. More or Less, BBC, April 8, 2020, https://www.bbc.co.uk/programmes/m000h6cb.

6. Author interview with Rebecca Goldin, December 12, 2017.

7. Paul J. C. Adachi and Teena Willoughby, “The Effect of Video Game Competition and Violence on Aggressive Behavior: Which Characteristic Has the Greatest Influence?,” Psychology of Violence 1, no. 4 (2011), 259–74, DOI: 10.1037/a0024908.

8. Steven Woolfe, “Immigration Post-Brexit,” Leave Means Leave research paper, https://static1.squarespace.com/static/5d08d13a2a109b000173bc76/t/5d47277f8fefae000124db10/1564944257844/FINAL-Low-Res-LML-Steven-Woolfe-migration-report-.pdf.

9. Jonathan Portes, “Who Are You Calling Low-Skilled?,” UK in a Changing Europe, April 12, 2017, https://ukandeu.ac.uk/who-are-you-calling-low-skilled/.

10. Robert Wright, “Brexit Visa Changes to Hit Sectors in Need of Low-Skilled Labour,” Financial Times, February 18, 2020, https://www.ft.com/content/890e84ce-5268-11ea-90ad-25e377c0ee1f.

11. Denis Campbell, “Fifth of 17 to 19-Year-Old Girls Self-Harm or Attempt Suicide,” Guardian, November 22, 2018, https://www.theguardian.com/society/2018/nov/22/concern-over-rise-in-suicide-attempts-among-young-women.

12. Mental Health of Children and Young People in England, 2017, NHS Digital, November 22, 2018, https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/2017/2017.

13. “Self-Harm,” NHS, page last reviewed May 25, 2018, https://www.nhs.uk/conditions/self-harm/.

14. Email correspondence with the NatCen Social Research (UK) press office, November 29, 2018.

15. Data from official sources such as the Office for National Statistics: https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/suicidesintheunitedkingdom/2017registrations#suicide-patterns-by-age.

16. Graeme Wearden, “Oxfam: 85 Richest People as Wealthy as Poorest Half of the World,” Guardian, January 20, 2014, https://www.theguardian.com/business/2014/jan/20/oxfam-85-richest-people-half-of-the-world. Nigel Morris, “World’s 85 Richest People Have as Much as Poorest 3.5 Billion,” Independent, January 20, 2014, https://www.independent.co.uk/news/world/politics/oxfam-warns-davos-of-pernicious-impact-of-the-widening-wealth-gap-9070714.html.

17. “Anatomy of a Killer Fact: The World’s 85 Richest People Own as Much as the Poorest 3.5 Billion,” From Poverty to Power (blog), January 31, 2014, https://oxfamblogs.org/fp2p/anatomy-of-a-killer-fact-the-worlds-85-richest-people-own-as-much-as-poorest-3-5-billion/. For the BBC interview with Fuentes, see “Do 85 Rich People Have Same Wealth as Half the World?,” March 19, 2014, https://www.bbc.com/news/magazine-26613682.

18. The underlying data come from the Global Wealth Report, which is published each year by Credit Suisse. The 2013 version supplied the data for Oxfam’s original “killer fact.”

19. On pensions as discussed in the footnote, see “Social Protection for Older Persons: Policy Trends and Statistics 2017–19,” International Labour Office, Social Protection Department (2018).

20. For the UK, the Institute for Fiscal Studies’ Living Standards, Poverty and Inequality in the UK. For global top incomes, the World Inequality Report. Another good source is Our World in Data. More specific references are provided in the notes below.

Rule Four: Step Back and Enjoy the View

1. For more reporting on this issue, listen to the June 8, 2018, episode of More or Less, presented by me and researched by my colleagues Richard Fenton-Smith and Richard Vadon, https://www.bbc.co.uk/programmes/p069jd0p.

2. Johan Galtung and Mari Holmboe Ruge, “The Structure of Foreign News: The Presentation of the Congo, Cuba and Cyprus Crises in Four Norwegian Newspapers,” Journal of Peace Research 2, no. 1 (1965), 64–90.

3. Max Roser, “Stop Saying That 2016 Was the Worst Year,” Washington Post, December 29, 2016, https://www.washingtonpost.com/posteverything/wp/2016/12/29/stop-saying-that-2016-was-the-worst-year/?utm_term=.bad894bad69a; see also NPR’s Planet Money, “The Fifty Year Newspaper,” December 29, 2017, https://www.npr.org/templates/transcript/transcript.php?storyId=574662798.

4. Colin P. Morice et al., “Quantifying Uncertainties in Global and Regional Temperature Change Using an Ensemble of Observational Estimates: The Hadcrut4 Dataset,” Journal of Geophysical Research (2012), 117, D08101, DOI: 10.1029/2011JD017187, describing data from the Met Office Hadley Centre. The data are charted by and downloadable from Our World in Data, https://ourworldindata.org/co2-and-other-greenhouse-gas-emissions. In the 1960s, global temperatures were typically around 0.1°C below the average of 1961–1990. In the twenty-first century they’ve typically been about 0.6°C above that average, and more recently above 0.7°C. Increase in temperatures, then, over the past fifty years, has been 0.7–0.8°C.

5. Max Roser, “The Short History of Global Living Conditions and Why It Matters That We Know It” (2018), published online at OurWorldInData.org, retrieved from https://ourworldindata.org/a-history-of-global-living-conditions-in-5-charts; for child mortality, Roser cites data from Gapminder and the World Bank.

6. See Figure E4 in the World Inequality Lab’s Executive Summary of the World Inequality Report 2018, https://wir2018.wid.world/files/download/wir2018-summary-english.pdf.

7. An excellent source is the Institute for Fiscal Studies Report on Living Standards, Poverty and Inequality in the UK. I’ve used the 2018 edition, the most recent available at the time of writing, https://www.ifs.org.uk/uploads/R145%20for%20web.pdf.

8. A good summary article on inequality around the world is on the Our World in Data website, written by Joe Hasell, an authority on the subject: https://ourworldindata.org/income-inequality-since-1990.

9. Author calculations, based on Natsal-3, the third National Survey of Sexual Attitudes and Lifestyles: http://timharford.com/2018/09/is-twitter-more-unequal-than-life-sex-or-happiness/.

10. Michael Blastland and Andrew Dilnot, The Tiger That Isn’t (London: Profile Books, 2008).

11. Andrew C. A. Elliott, Is That a Big Number? (Oxford: Oxford University Press, 2018).

12. Tali Sharot, “The Optimism Bias,” TED Talk, 2012, https://www.ted.com/talks/tali_sharot_the_optimism_bias/transcript#t-18026.

13. Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus and Giroux, 2010).

14. Ross A. Miller and Karen Albert, “If It Leads, It Bleeds (and If It Bleeds, It Leads): Media Coverage and Fatalities in Militarized Interstate Disputes,” Political Communication 32, no. 1 (2015), 61–82, DOI: 10.1080/10584609.2014.880976; Barbara Combs and Paul Slovic, “Newspaper Coverage of Causes of Death,” Journalism Quarterly 56, no. 4 (1979), 837–43, 849.

15. “Smoking and Tobacco Use, Fast Facts,” Centers for Disease Control and Prevention, page last reviewed May 21, 2020, https://www.cdc.gov/tobacco/data_statistics/fact_sheets/fast_facts/. There are 1,300 deaths a day from smoking-related diseases, about 40,000 a month; almost 3,000 people were killed in the attacks of September 11, 2001.

16. “Steven Pinker on the Case for Optimism,” interview by Chris Anderson, TED2018 conference, https://www.ted.com/talks/the_ted_interview_steven_pinker_on_why_our_pessimism_about_the_world_is_wrong/transcript?language=en.

17. Steven Pinker mentions in the endnotes of Enlightenment Now (New York: Viking, 2018) that this correspondence took place in 1982.

18. Quoted in Denis Campbell, “Stroke Association Warns of Alarming Rise in Number of Victims,” Guardian, May 12, 2015, https://www.theguardian.com/society/2015/may/12/stroke-association-warns-of-alarming-rise-in-number-of-victims; see also More or Less, May 17, 2015, with the analysis of this claim: https://www.bbc.co.uk/programmes/b05tpz78.

19. “Public Pessimism and Misunderstanding Undermining Fight against Global Poverty,” Oxfam press release, September 22, 2016, http://oxfamapps.org/media/ppdwr.

20. A useful survey of various relevant graphs is Max Roser and Mohamed Nagdy, “Optimism and Pessimism,” published online at OurWorldInData.org, 2018, retrieved from https://ourworldindata.org/optimism-pessimism, particularly Section I.1 with graphs from Eurobarometer and Ipsos Mori.

21. Martyn Lewis, “Not My Idea of Good News,” Independent, April 26, 1993, https://www.independent.co.uk/voices/not-my-idea-of-good-news-at-the-end-of-a-week-of-horrifying-events-martyn-lewis-bbc-presenter-argues-1457539.html.

22. Roser, “The Short History of Global Living Conditions and Why It Matters That We Know It”; underlying data from the World Bank and from Bourguignon and Morrisson, “Inequality among World Citizens: 1820–1992,” American Economic Review 92, no. 4 (2002), 727–48. In 1993 there were 1.94 billion people living in extreme poverty; by 2015 that had fallen to 0.7 billion (705.55 million). The rate of improvement averages 153,600 a day, although of course we have no way of measuring the daily rate as it fluctuates.

23. Samantha Vanderslott, Bernadeta Dadonaite, and Max Roser, “Vaccination,” published online at OurWorldInData.org (2020), retrieved from https://ourworldindata.org/vaccination.

24. Anna Rosling Rönnlund, Hans Rosling, and Ola Rosling, Factfulness (New York: Flatiron, 2018).

25. Gillian Tett, “Silos and Silences,” Derivatives: Financial Innovation and Stability, Banque de France Financial Stability Review 14 (July 2010), https://publications.banque-france.fr/sites/default/files/medias/documents/financial-stability-review-14_2010-07.pdf.

26. Rolf Dobelli, “News Is Bad for You—and Giving Up Reading It Will Make You Happier,” Guardian, April 12, 2013, https://www.theguardian.com/media/2013/apr/12/news-is-bad-rolf-dobelli.

27. Nassim Nicholas Taleb, The Bed of Procrustes (London: Penguin Books, 2010).

28. Bill Hanage and Mark Lipsitch, “How to Report on the COVID-19 Outbreak Responsibly,” Scientific American, February 23, 2020, https://blogs.scientificamerican.com/observations/how-to-report-on-the-covid-19-outbreak-responsibly/.

Rule Five: Get the Backstory

1. Sheena Iyengar and Mark Lepper, “When Choice Is Demotivating: Can One Desire Too Much of a Good Thing?,” Journal of Personality and Social Psychology 79, 2000.

2. Author interview with Benjamin Scheibehenne, October 2009. (I’d like to claim I was ahead of the curve on this one.)

3. B. Scheibehenne, R. Greifeneder, and P. M. Todd, “Can There Ever Be Too Many Options? A Meta-Analytic Review of Choice Overload,” Journal of Consumer Research 37 (2010), 409–25, http://scheibehenne.de/ScheibehenneGreifenederTodd2010.pdf.

4. Nicole Lyn Pesce, “Ten Kickstarter Products That Raised the Most Money,” MarketWatch, June 22, 2017, https://www.marketwatch.com/story/10-kickstarter-products-that-raised-the-most-money-2017-06-22-10883052.

5. The story is well told in Jordan Ellenberg’s book How Not to Be Wrong (New York: Penguin, 2014), with the relevant extract here: “Abraham Wald and the Missing Bullet Holes,” Medium, July 14, 2016, https://medium.com/@penguinpress/an-excerpt-from-how-not-to-be-wrong-by-jordan-ellenberg-664e708cfc3d.

6. A technical summary (along with some grumbling about how the story has been exaggerated) is in Bill Casselman, “The Legend of Abraham Wald,” American Mathematical Society, http://www.ams.org/publicoutreach/feature-column/fc-2016-06.

7. An excellent account of the controversy is Daniel Engber, “Daryl Bem Proved ESP Is Real Which Means Science Is Broken,” Slate, May 17, 2017, https://slate.com/health-and-science/2017/06/daryl-bem-proved-esp-is-real-showed-science-is-broken.html.

8. Chris French, “Precognition Studies and the Curse of the Failed Replications,” Guardian, March 15, 2012, https://www.theguardian.com/science/2012/mar/15/precognition-studies-curse-failed-replications.

9. Nosek was speaking on the Planet Money podcast, episode 677, “The Experiment,” https://www.npr.org/sections/money/2018/03/07/591213302/episode-677-the-experiment-experiment.

10. Brian Nosek has given useful interviews to several podcasts, including You Are Not So Smart (episode 100), https://youarenotsosmart.com/2017/07/19/yanss-100-the-replication-crisis/; Planet Money (episode 677); EconTalk, November 16, 2015, http://www.econtalk.org/brian-nosek-on-the-reproducibility-project/; The Hidden Brain (episode 32), https://www.npr.org/templates/transcript/transcript.php?storyId=477921050; as well as BBC Analysis, “The Replication Crisis,” November 12, 2018, https://www.bbc.co.uk/programmes/m00013p9.

11. This figure of thirty-nine is based on the subjective opinion of the replicating researchers. Did their results basically back up the original study, or not? That’s a judgment call. An alternative metric is to ask how many of the replication studies produced results that passed the standard (but rather problematic) hurdle of “statistical significance.” Only thirty-six did; ninety-seven of the original studies had cleared that hurdle. Open Science Collaboration, “Estimating the Reproducibility of Psychological Science,” Science 28, no. 6251 (August 2015), 349, DOI: 10.1126/science.aac4716.

 12. Brief film on YouTube: “Derren Brown—Ten Heads in a Row,” ThinkSceptically, April 8, 2012, https://www.youtube.com/watch?v=n1SJ-Tn3bcQ.

13. Planet Money, episode 677.

14. F. J. Anscombe, “Fixed-Sample-Size Analysis of Sequential Observations,” Biometrics 10, no. 1 (1954), 89–100, JSTOR, www.jstor.org/stable/3001665; and Andrew Gelman’s blog post “Statistical Inference, Modeling and Social Science,” May 2, 2018, https://statmodeling.stat.columbia.edu/2018/05/02/continuously-increased-number-animals-statistical-significance-reached-support-conclusions-think-not-bad-actually/.

15. David J. Hand, Dark Data (Princeton, NJ: Princeton University Press, 2020).

16. Andrew Gelman and Eric Loken, “The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No ‘Fishing Expedition’ or ‘P-Hacking’ and the Research Hypothesis Was Posited Ahead of Time,” working paper, November 14, 2013, http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf.

17. J. P. Simmons, L. D. Nelson, and U. Simonsohn, “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Psychological Science 22, no. 11 (2011), 1359–66, https://doi.org/10.1177/0956797611417632.

18. Kai Kupferschmidt, “More and More Scientists Are Preregistering Their Studies. Should You?,” Science, September 21, 2018.

19. Anjana Ahuja, “Scientists Strike Back against Statistical Tyranny,” Financial Times, March 27, 2019, https://www.ft.com/content/36f9374c-5075-11e9-8f44-fe4a86c48b33.

20. Darrell Huff, How to Lie with Statistics (New York: W. W. Norton, 1993), 40.

21. John Ioannidis, “Why Most Published Research Findings Are False,” PLoS Medicine 2, no. 8 (August 2005), e124, DOI: 10.1371/journal.pmed.0020124, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/.

22. R. F. Baumeister et al., “Ego Depletion: Is the Active Self a Limited Resource?,” Journal of Personality and Social Psychology 74, no. 5 (1998), 1252–65, http://dx.doi.org/10.1037/0022-3514.74.5.1252; “The End of Ego Depletion Theory?,” Neuroskeptic (blog), July 31, 2016, http://blogs.discovermagazine.com/neuroskeptic/2016/07/31/end-of-ego-depletion/#.XGGyflz7SUk.

23. Amy Cuddy, “Your Body Language May Shape Who You Are,” TED Talk, 2012, https://www.ted.com/talks/amy_cuddy_your_body_language_shapes_who_you_are/transcript?language=en.

24. Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus and Giroux, 2010), 53–57.

25. Ed Yong, “Nobel Laureate Challenges Psychologists to Clean Up Their Act,” Nature News, October 3, 2012, https://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535.

26. Ben Goldacre, “Backwards Step on Looking into the Future,” Guardian, April 23, 2011, https://www.theguardian.com/commentisfree/2011/apr/23/ben-goldacre-bad-science.

27. Robin Wrigglesworth, “How a Herd of Cows Trampled on Human Stockpickers,” Financial Times, January 21, 2020, https://www.ft.com/content/563d61dc-3b70-11ea-a01a-bae547046735?.

28. Burton Malkiel, “Returns from Investing in Equity Funds,” working paper, Princeton University, 1994; published as “Returns from Investing in Equity Funds: 1971–1991,” Journal of Finance 50 (1995), 549–72.

 29. Eric Balchunas, “How the Vanguard Effect Adds Up to $1 Trillion,” Bloomberg .com, August 30, 2016, https://www.bloomberg.com/opinion/articles/2016-08-30/how-much-has-vanguard-saved-investors-try-1-trillion.

30. For an accessible overview, see Ben Goldacre, “What Doctors Don’t Know about the Drugs They Prescribe,” TED Talk, 2012, https://www.ted.com/talks/ben_goldacre_what_doctors_don_t_know_about_the_drugs_they_prescribe/footnotes?language=en.

31. Erick Turner et al., “Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy,” New England Journal of Medicine, January 17, 2008, https://www.nejm.org/doi/full/10.1056/NEJMsa065779.

32. Ben Goldacre, “Transparency, Beyond Publication Bias,” talk given at the International Journal of Epidemiology Conference, 2016; available at https://www.badscience.net/2016/10/transparency-beyond-publication-bias-a-video-of-my-super-speedy-talk-at-ije/.

33. Ben Goldacre et al., “COMPare: A Prospective Cohort Study Correcting and Monitoring 58 Misreported Trials in Real Time,” Trials 20, no. 118 (2019), https://doi.org/10.1186/s13063-019-3173-2.

34. Ben Goldacre, “Transparency, Beyond Publication Bias.”

35. Amy Sippett, “Does the Backfire Effect Exist?,” Full Fact (blog), March 20, 2019, https://fullfact.org/blog/2019/mar/does-backfire-effect-exist/; Brendan Nyhan, “Read this! Widespread neglect of more recent studies since my paper w/ @JasonReifler,” Twitter, March 20, 2019, 10:40 a.m., https://twitter.com/BrendanNyhan/status/1108377656414879744.

36. Author interview with Richard Thaler, July 17, 2019.

37. “The Replication Crisis,” BBC Analysis, November 12, 2018, https://www.bbc.co.uk/programmes/m00013p9.

38. Antonio Granado, “Slaves to Journals, Serfs to the Web: The Use of the Internet in Newsgathering among European Science Journalists,” Journalism 12, no. 7 (2011), 794–813.

39. A. L. Cochrane, “Sickness in Salonica: My First, Worst, and Most Successful Clinical Trial,” British Medical Journal (Clinical Research Edition) 289, no. 6460 (1984): 1726–27, DOI: 10.1136/bmj.289.6460.1726.

40. Cochrane Community, “A Brief History of Cochrane,” https://community.cochrane.org/handbook-sri/chapter-1-introduction/11-cochrane/112-brief-history-cochrane [inactive].

41. Alan Mozes, “Yoga May Be Right Move Versus Urinary Incontinence,” WebMD, May 22, 2018, https://www.webmd.com/urinary-incontinence-oab/news/20180522/yoga-may-be-right-move-versus-urinary-incontinence#1.

42. Emma Innes, “Could Yoga Cure Incontinence?,” DailyMail.com, May 12, 2014, https://www.dailymail.co.uk/health/article-2626209/Could-yoga-cure-INCONTINENCE-Exercise-strengthens-pelvic-floor-muscles-reducing-leakage.html.

43. “Yoga and Incontinence,” Home Care Delivered (blog), https://www.hcd.com/incontinence/yoga-incontinence/.

44. Alison J. Huang et al., “A Group-Based Yoga Therapy Intervention for Urinary Incontinence in Women,” Female Pelvic Medicine & Reconstructive Surgery 20, no. 3 (2014), 147–54, DOI: 10.1097/SPV.0000000000000072, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4310548/.

45. L. S. Wieland et al., “Yoga for Treating Urinary Incontinence in Women,” Cochrane Database of Systematic Reviews 2019 2, art. no. CD012668, DOI: 10.1002/14651858.CD012668.pub2.

Rule Six: Ask Who Is Missing

1. Rod Bond and Peter B. Smith, “Culture and Conformity: A Meta-Analysis of Studies Using Asch’s (1952b, 1956) Line Judgment Task,” Psychological Bulletin 119, no. 1 (1996), 111–37, http://dx.doi.org/10.1037/0033-2909.119.1.111.

2. Tim Harford, “The Truth about Our Norm-Core,” Financial Times, June 12, 2015, http://timharford.com/2015/06/the-truth-about-our-norm-core/.

3. Bond and Smith, “Culture and Conformity”; and Natalie Frier et al., “The Effects of Group Conformity Based on Sex,” Celebrating Scholarship and Creativity Day, Paper 83 (2016), http://digitalcommons.csbsju.edu/elce_cscday/83.

4. Tim Harford, “Trump, Brexit and How Politics Loses the Capacity to Shock,” Financial Times, November 16, 2018, https://www.ft.com/content/b730c95c-e82e-11e8-8a85-04b8afea6ea3.

5. Caroline Criado Perez, Invisible Women (London: Chatto & Windus, 2019); the interview was broadcast on BBC Radio 4 on May 17, 2019, and is available on the More or Less website: https://www.bbc.co.uk/programmes/m00050rd.

6. Peter Hofland, “Reversal of Fortune,” Onco’Zine, November 30, 2013, https://oncozine.com/reversal-of-fortune-how-a-vilified-drug-became-a-life-saving-agent-in-the-war-against-cancer/.

7. R. Dmitrovic, A. R. Kunselman, and R. S. Legro, “Sildenafil Citrate in the Treatment of Pain in Primary Dysmenorrhea: A Randomized Controlled Trial,” Human Reproduction 28, no. 11 (November 2013): 2958–65, DOI: 10.1093/humrep/det324, Epub August 6, 2013, https://www.ncbi.nlm.nih.gov/pubmed/23925396?dopt=Abstract.

8. BBC, “Supermarket Stockpiling, A-Level Results and Covid-19 Gender Disparity,” More or Less podcast, March 31, 2020, https://www.bbc.co.uk/sounds/play/m000h7st.

9. Mayra Buvinic and Ruth Levine, “Closing the Gender Data Gap,” Significance, April 8, 2016, and Charlotte McDonald, “Is There a Sexist Data Crisis?,” BBC News, May 18, 2016, https://www.bbc.co.uk/news/magazine-36314061.

10. Shelly Lundberg, Robert Pollak, and Terence J. Wales, “Do Husbands and Wives Pool Their Resources? Evidence from the United Kingdom Child Benefit,” Journal of Human Resources 32, no. 3 (1997), 463–80.

11. Mayra Buvinic and Ruth Levine, “Closing the Gender Data Gap.”

12. Suzannah Brecknell, “Interview: Full Fact’s Will Moy on Lobbyist ‘Nonsense,’ Official Corrections and Why We Know More about Golf Than Crime Stats,” Civil Service World, May 5, 2016, https://www.civilserviceworld.com/articles/interview/interview-full-fact%E2%80%99s-will-moy-lobbyist-%E2%80%9Cnonsense%E2%80%9D-official-corrections-and-why.

13. Maurice C. Bryson, “The Literary Digest Poll: Making of a Statistical Myth,” American Statistician 30, no. 4 (1976), 184–85, DOI: 10.1080/00031305.1976.10479173; and Peverill Squire, “Why the 1936 Literary Digest Poll Failed,” Public Opinion Quarterly 52, no. 1 (1988), 125–33, JSTOR, www.jstor.org/stable/2749114.

14. P. Whiteley, “Why Did the Polls Get It Wrong in the 2015 General Election? Evaluating the Inquiry into Pre-Election Polls,” Political Quarterly 87 (2016), 437–42, DOI: 10.1111/1467-923X.12274.

15. John Curtice, “Revealed: Why the Polls Got It So Wrong in the British General Election,” The Conversation, January 14, 2016, https://theconversation.com/revealed-why-the-polls-got-it-so-wrong-in-the-british-general-election-53138.

 16. Nate Cohn, “A 2016 Review: Why Key State Polls Were Wrong about Trump,” New York Times, May 31, 2017, https://www.nytimes.com/2017/05/31/upshot/a-2016-review-why-key-state-polls-were-wrong-about-trump.html; and Andrew Mercer, Claudia Deane, and Kyley McGeeney, “Why 2016 Election Polls Missed Their Mark,” Pew Research Fact Tank blog, November 9, 2015, http://www.pewresearch.org/fact-tank/2016/11/09/why-2016-election-polls-missed-their-mark/.

17. “2011 Census Statistics for England and Wales: March 2011 QMI,” Office for National Statistics, last updated May 16, 2013, https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/methodologies/2011censusstatisticsforenglandandwalesmarch2011qmi.

18. Author interview with Viktor Mayer-Schönberger, March 2014.

19. “Social Media Factsheet,” Pew Research Center, research conducted January 2018, https://www.pewinternet.org/fact-sheet/social-media/.

20. Kate Crawford, “The Hidden Biases in Big Data,” Harvard Business Review, April 1, 2013, https://hbr.org/2013/04/the-hidden-biases-in-big-data.

21. Leon Kelion, “Coronavirus: Covid-19 Detecting Apps Face Teething Problems,” BBC News, April 8, 2020, https://www.bbc.co.uk/news/technology-52215290.

22. Kate Crawford, “Artificial Intelligence’s White Guy Problem,” New York Times, June 25, 2016, https://www.nytimes.com/2016/06/26/opinion/sunday/artificial-intelligences-white-guy-problem.html.

Rule Seven: Demand Transparency When the Computer Says No

1. Jeremy Ginsberg et al., “Detecting Influenza Epidemics Using Search Engine Query Data,” Nature 457, no. 7232 (February 19, 2009), 1012–14, DOI: 10.1038/nature07634.

2. Parts of this chapter are closely based on my article “Big Data: Are We Making a Big Mistake?,” Financial Times, March 28, 2014, https://www.ft.com/content/21a6e7d8-b479-11e3-a09a-00144feabdc0. I interviewed David Hand, Kaiser Fung, Viktor Mayer-Schönberger, David Spiegelhalter, and Patrick Wolfe in early 2014 for the piece.

3. David Lazer and Ryan Kennedy, “What We Can Learn from the Epic Failure of Google Flu Trends,” Wired, October 1, 2015, https://www.wired.com/2015/10/can-learn-epic-failure-google-flu-trends/; and Declan Butler, “What Google Flu Got Wrong,” Nature, February 13, 2013, https://www.nature.com/news/when-google-got-flu-wrong-1.12413.

4. Google Flu Trends and Google Dengue Trends data pages, updated August 2014, https://www.google.org/flutrends/about/ [inactive].

5. D. Lazer et al., “The Parable of Google Flu: Traps in Big Data Analysis,” Science 343, no. 6176 (March 2014), 1203–5.

6. Samantha Cook et al., “Assessing Google Flu Trends Performance in the United States during the 2009 Influenza Virus A (H1N1) Pandemic,” PLoS ONE 6, no. 8: e23610 (2011), https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0023610.

7. Janelle Shane, You Look Like a Thing and I Love You (New York: Little, Brown, 2019).

8. For comprehensive reporting of this affair, see “The Cambridge Analytica Files,” on the Observer/Guardian website, https://www.theguardian.com/news/series/cambridge-analytica-files.

 9. Charles Duhigg, “How Companies Learn Your Secrets,” New York Times Magazine, February 19, 2012, https://www.nytimes.com/2012/02/19/magazine/shopping-habits.html.

10. Hannah Fry, Hello World: Being Human in the Age of Computers (London: W. W. Norton, 2018).

11. Cathy O’Neil, Weapons of Math Destruction (London: Allen Lane, 2016).

12. Freakonomics, radio episode 268: “Bad Medicine, Part 1,” August 16, 2017, http://freakonomics.com/podcast/bad-medicine-part-1-story-rebroadcast/.

13. P. A. Mackowiak, S. S. Wasserman, and M. M. Levine, “A Critical Appraisal of 98.6°F, the Upper Limit of the Normal Body Temperature, and Other Legacies of Carl Reinhold August Wunderlich,” JAMA 268, no. 12 (1992), 1578–80, DOI: 10.1001/jama.1992.03490120092034.

14. Jeffrey Dastin, “Amazon Scraps Secret AI Recruiting Tool That Showed Bias against Women,” Reuters, October 10, 2018, https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G.

15. Gerd Gigerenzer and Stephanie Kurzenhaeuser, “Fast and Frugal Heuristics in Medical Decision Making,” in Roger Bibace et al., Science and Medicine in Dialogue: Thinking through Particulars and Universals (Westport, CT: Praeger, 2005), 3–15.

16. Paul Meehl, Clinical vs. Statistical Prediction (Minneapolis: University of Minnesota Press, 1954).

17. Fry, Hello World.

18. Mandeep K. Dhami and Peter Ayton, “Bailing and Jailing the Fast and Frugal Way,” Journal of Behavioral Decision Making 14, no. 2 (2001).

19. The original draft of this chapter, like most of the articles I cite, used words such as “commit further crimes” and “reoffend.” But as Cathy O’Neil pointed out to me, what is usually being predicted is being arrested again. Not everyone who is arrested has committed a crime, and not everyone who commits a crime is arrested. I (and my sources) should have been more careful, per my advice under rule three.

20. Jon Kleinberg et al., “Human Decisions and Machine Predictions,” Quarterly Journal of Economics 133, no. 1 (February 2018), 237–93, https://doi.org/10.1093/qje/qjx032; see also Cass R. Sunstein, “Algorithms, Correcting Biases,” working paper, December 12, 2018.

21. David Jackson and Gary Marx, “Data Mining Program Designed to Predict Child Abuse Proves Unreliable, DCFS Says,” Chicago Tribune, December 6, 2017; and Dan Hurley, “Can an Algorithm Tell When Kids Are in Danger?,” New York Times Magazine, January 2, 2018, https://www.nytimes.com/2018/01/02/magazine/can-an-algorithm-tell-when-kids-are-in-danger.html.

22. Hurley, “Can an Algorithm.”

23. Andrew Gelman, “Flaws in Stupid Horrible Algorithm Revealed Because It Made Numerical Predictions,” Statistical Modeling, Causal Inference, and Social Science (blog), July 3, 2018, https://statmodeling.stat.columbia.edu/2018/07/03/flaws-stupid-horrible-algorithm-revealed-made-numerical-predictions/.

24. Sabine Hossenfelder, “Blaise Pascal, Florin Périer, and the Puy de Dôme Experiment,” BackRe(Action) (blog), November 21, 2007, http://backreaction.blogspot.com/2007/11/blaise-pascal-florin-p-and-puy-de-d.html; and David Wootton, The Invention of Science: A New History of the Scientific Revolution (London: Allen Lane, 2015), chap. 8.

 25. See, for example, Louis Trenchard More, “Boyle as Alchemist,” Journal of the History of Ideas 2, no. 1 (January 1941), 61–76; and Adam Mann, “The Strange, Secret History of Isaac Newton’s Papers,” Q&A with Sarah Dry, Wired, May 14, 2014, https://www.wired.com/2014/05/newton-papers-q-and-a/.

26. Wootton, The Invention of Science, 340.

27. James Burke, Connections (Boston: Little, Brown, 1978; repr. 1995), 74.

28. Wootton, The Invention of Science, 357.

29. Jeff Larson et al., “How We Analyzed the COMPAS Recidivism Algorithm,” ProPublica, May 23, 2016, https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm.

30. Sam Corbett-Davies et al., “Algorithmic Decision Making and the Cost of Fairness,” Cornell University, arXiv:1701.08230; and Sam Corbett-Davies et al., “A Computer Program Used for Bail and Sentencing Decisions Was Labeled Biased against Blacks: It’s Actually Not That Clear,” Washington Post, October 17, 2016, https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/17/can-an-algorithm-be-racist-our-analysis-is-more-cautious-than-propublicas/.

31. Ed Yong, “A Popular Algorithm Is No Better at Predicting Crimes Than Random People,” The Atlantic, January 17, 2018, https://www.theatlantic.com/technology/archive/2018/01/equivant-compas-algorithm/550646/.

32. Yong, “A Popular Algorithm.”

33. Julia Dressel and Hany Farid, “The Accuracy, Fairness and Limits of Predicting Recidivism,” Science Advances 4, no. 1 (January 17, 2018), DOI: 10.1126/sciadv.aao5580.

34. Onora O’Neill’s “Reith Lectures: A Question of Trust” (BBC Radio 4, 2002, http://www.bbc.co.uk/radio4/reith2002/) and TED Talk “What We Don’t Understand about Trust” (September 2013, https://www.ted.com/speakers/onora_o_neill) are both well worth listening to. Themes of intelligent openness are explored in depth in the Royal Society report Science as an Open Enterprise (2012), of which O’Neill was an author. In his book The Art of Statistics (New York: Basic Books, 2019), David Spiegelhalter shows how O’Neill’s principles can be applied to evaluating algorithms.

35. Author email interview with Cathy O’Neil, August 29, 2019.

36. Jack Nicas, “How YouTube Drives Viewers to the Internet’s Darkest Corners,” Wall Street Journal, February 7, 2018, https://www.wsj.com/articles/how-youtube-drives-viewers-to-the-internets-darkest-corners-1518020478; and Zeynep Tufekci, “YouTube, the Great Radicalizer,” New York Times, March 10, 2018, https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-radical.html. But see in contrast Mark Ledwich and Anna Zaitsev, “Algorithmic Extremism: Examining YouTube’s Rabbit Hole of Radicalization,” Cornell University, arXiv:1912.11211, https://arxiv.org/abs/1912.11211.

37. Ryan Singal, “Netflix Spilled Your Brokeback Mountain Secret, Lawsuit Claims,” Wired, December 17, 2009, https://www.wired.com/2009/12/netflix-privacy-lawsuit/; and Blake Hallinan and Ted Striphas, “Recommended for You: The Netflix Prize and the Production of Algorithmic Culture,” New Media and Society, 2016, https://journals.sagepub.com/doi/pdf/10.1177/1461444814538646.

Rule Eight: Don’t Take Statistical Bedrock for Granted

1. This is a translation of a Danish TV interview, discussed in Peter Vinthagen Simpson, “Hans Rosling: ‘You Can’t Trust the Media,’” The Local, September 5, 2015, https://www.thelocal.se/20150905/hans-rosling-you-cant-trust-the-media.

2. Laura Smith, “In 1974, a Stripper Known as the ‘Tidal Basin Bombshell’ Took Down the Most Powerful Man in Washington,” Timeline, September 18, 2017, https://timeline.com/wilbur-mills-tidal-basin-3c29a8b47ad1; Stephen Green and Margot Hornblower, “Mills Admits Being Present during Tidal Basin Scuffle,” Washington Post, October 11, 1974.

3. “The Stripper and the Congressman: Fanne Foxe’s Story,” Rialto Report Podcast, episode 82.

4. Alice M. Rivlin, “The 40th Anniversary of the Congressional Budget Office,” Brookings: On the Record, March 2, 2015, https://www.brookings.edu/on-the-record/40th-anniversary-of-the-congressional-budget-office/.

5. Philip Joyce, “The Congressional Budget Office at Middle Age,” Hutchins Center at Brookings, Working Paper #9, February 17, 2015.

6. Quoted in Nancy D. Kates, Starting from Scratch: Alice Rivlin and the Congressional Budget Office (Cambridge, MA: John F. Kennedy School of Government, Harvard University, 1989).

7. Elaine Povich, “Alice Rivlin, Budget Maestro Who ‘Helped Save Washington’ in Fiscal Crisis, Dies at 88,” Washington Post, May 14, 2019, https://www.washingtonpost.com/local/obituaries/alice-rivlin-budget-maestro-who-helped-save-washington-in-fiscal-crisis-dies-at-88/2019/05/14/c141c996-0ff9-11e7-ab07-07d9f521f6b5_story.html.

8. Andrew Prokop, “The Congressional Budget Office, Explained,” Vox, June 26, 2017, https://www.vox.com/policy-and-politics/2017/3/13/14860856/congressional-budget-office-cbo-explained.

9. John Frendreis and Raymond Tatalovich, “Accuracy and Bias in Macroeconomic Forecasting by the Administration, the CBO, and the Federal Reserve Board,” Polity 32, no. 4 (2000), 623–32, DOI: 10.2307/3235295; Holly Battelle, CBO’s Economic Forecasting Record (Washington, DC: Congressional Budget Office, 2010); Committee for a Responsible Federal Budget, “Hindsight Is 2020: A Look Back at CBO’s Economic Forecasting,” January 2013, https://www.crfb.org/blogs/hindsight-2020-look-back-cbos-economic-forecasting.

10. Office for Budget Responsibility, Forecast Evaluation Report 2019, December 2019, https://obr.uk/docs/dlm_uploads/Forecast_evaluation_report_December_2019-1.pdf.

11. Malcolm Bull, “Can the Poor Think?,” London Review of Books 41, no. 13 (July 4, 2019).

12. Bourree Lam, “After a Good Jobs Report, Trump Now Believes Economic Data,” The Atlantic, March 10, 2017, https://www.theatlantic.com/business/archive/2017/03/trump-spicer-jobs-report/519273/.

13. Esther King, “Germany Records Lowest Crime Rate since 1992,” Politico, May 8, 2017, https://www.politico.eu/article/germany-crime-rate-lowest-since-1992/.

14. For discussion and the full Trump tweets, see Matthew Yglesias, “Trump Just Tweeted That ‘Crime in Germany Is Way Up.’ It’s Actually at Its Lowest Level Since 1992,” Vox, June 18, 2018; and Christopher F. Schuetze and Michael Wolgelenter, “Fact Check: Trump’s False and Misleading Claims about Germany’s Crime and Immigration,” New York Times, June 18, 2018.

15. Diane Coyle, GDP: A Brief but Affectionate History (Princeton, NJ: Princeton University Press, 2014), 3–4.

 16. European Commission, Report on Greek Government Deficit and Debt Statistics, January 2010, https://ec.europa.eu/eurostat/documents/4187653/6404656/COM_2010_report_greek/c8523cfa-d3c1-4954-8ea1-64bb11e59b3a.

17. Beat Balzli, “Greek Debt Crisis: How Goldman Sachs Helped Greece to Mask Its True Debt,” Der Spiegel, February 8, 2010, https://www.spiegel.de/international/europe/greek-debt-crisis-how-goldman-sachs-helped-greece-to-mask-its-true-debt-a-676634.html.

18. The International Statistical Institute has a chronological account of the sorry tale, last updated by G. O’Hanlon and H. Snorrason, July 2018: “Court Proceedings against Andreas Georgiou,” https://isi-web.org/images/news/2018-07_Court-proceedings-against-Andreas-Georgiou.pdf.

19. “Commendation of Andreas Georgiou,” press release, International Statistical Association, September 18, 2018, https://www.isi-web.org/images/2018/Press%20release%20Commendation%20for%20Andreas%20Georgiou%20Aug%202018.pdf.

20. R. Langkjær-Bain, “Trials of a Statistician,” Significance 14 (2017), 14–19, DOI: 10.1111/j.1740-9713.2017.01052.x; “An Augean Stable,” The Economist, February 13, 2016, https://www.economist.com/the-americas/2016/02/13/an-augean-stable; “The Price of Cooking the Books,” The Economist, February 25, 2012, https://www.economist.com/the-americas/2012/02/25/the-price-of-cooking-the-books.

21. Langkjær-Bain, “Trials of a Statistician.”

22. Author interview with Denise Lievesley, July 2, 2018.

23. “Tanzania Law Punishing Critics of Statistics ‘Deeply Concerning’: World Bank,” Reuters, October 3, 2018, https://www.reuters.com/article/us-tanzania-worldbank/tanzania-law-punishing-critics-of-statistics-deeply-concerning-world-bank-idUSKCN1MD17P.

24. Amy Kamzin, “Dodgy Data Makes It Hard to Judge Modi’s Job Promises,” Financial Times, October 8, 2018, https://www.ft.com/content/1a008ebe-cad4-11e8-9fe5-24ad351828ab.

25. Steven Chase and Tavia Grant, “Statistics Canada Chief Falls on Sword over Census,” Globe and Mail, July 21, 2010, https://www.theglobeandmail.com/news/politics/statistics-canada-chief-falls-on-sword-over-census/article1320915/.

26. Langkjær-Bain, “Trials of a Statistician.”

27. Nicole Acevedo, “Puerto Rico Faces Lawsuits over Hurricane Death Count Data,” NBC News, June 1, 2018; and Joshua Barajas, “Hurricane Maria’s Official Death Toll Is 46 Times Higher Than It Was Almost a Year Ago. Here’s Why,” PBS NewsHour, August 30, 2018, https://www.pbs.org/newshour/nation/hurricane-marias-official-death-toll-is-46-times-higher-than-it-was-almost-a-year-ago-heres-why.

28. Office for National Statistics, 2011 Census Benefits Evaluation Report, last updated October 9, 2017, https://www.ons.gov.uk/census/2011census/2011censusbenefits/2011censusbenefitsevaluationreport#unquantified-benefits; Ian Cope, “The Value of Census Statistics,” Office for National Statistics, https://www.ukdataservice.ac.uk/media/455474/cope.pdf. http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/.

29. Carl Bakker, Valuing the Census (Wellington: Statistics New Zealand, 2014), https://www.stats.govt.nz/assets/Research/Valuing-the-Census/valuing-the-census.pdf.

30. Mónica I. Feliú-Mójer, “Why Is Puerto Rico Dismantling Its Institute of Statistics?,” Scientific American: Voices (blog), February 1, 2018.

 31. Testimony, CBO’s Appropriation Request for Fiscal Year 2020, Subcommittee on the Legislative Branch, Committee on Appropriations, U.S. House of Representatives, CBO director Keith Hill testimony, February 26, 2019, https://www.cbo.gov/publication/54965.

32. Ellen Hughes-Cromwick and Julia Coronado, “The Value of US Government Data to US Business Decisions,” Journal of Economic Perspectives 33, no. 1 (2019), 131–46, DOI: 10.1257/jep.33.1.131.

33. Milton and Rose Friedman, Two Lucky People (Chicago: University of Chicago Press, 1998), quoted in Neil Monnery, “Hong Kong’s Postwar Transformation Shows How Fewer Data Can Sometimes Boost Growth,” blog post, London School of Economics and Political Science, June 30, 2017, https://blogs.lse.ac.uk/businessreview/2017/06/30/hong-kongs-postwar-transformation-shows-how-fewer-data-can-sometimes-boost-growth/.

34. James C. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed (New Haven: Yale University Press, 1998).

35. Perry Link, “China: From Famine to Oslo,” New York Review of Books, January 13, 2011.

36. For a discussion of the death toll under Stalin, see Timothy Snyder, “Hitler vs. Stalin: Who Killed More?,” New York Review of Books, March 10, 2011—a more sensitively written piece than the title suggests. For more on the 1937 census, see Daniel Sandford, “In Moscow, History Is Everywhere,” BBC News, November 2, 2012; and Catherine Merridale, “The 1937 Census and the Limits of Stalinist Rule,” Historical Journal 39, no. 1 (1996); and “Called to Account,” The Economist, September 3, 2016, https://www.economist.com/finance-and-economics/2016/09/03/called-to-account.

37. Catherine Merridale, “The 1937 Census and the Limits of Stalinist Rule.”

38. Adam Tooze, Statistics and the German State, 1900–1945 (Cambridge, UK: Cambridge University Press, 2001), 257.

39. Author interview with Denise Lievesley, March 11, 2019.

40. Hetan Shah, “How to Save Statistics from the Threat of Populism,” Financial Times, October 21, 2018, https://www.ft.com/content/ca491f18-d383-11e8-9a3c-5d5eac8f1ab4.

41. Nicholas Eberstadt et al., “‘In Order That They Might Rest Their Arguments on Fact’: The Vital Role of Government-Collected Data,” AEI/Hamilton Project report, March 2017.

42. For more on the review, see G. Hoinville and T. M. F. Smith, “The Rayner Review of Government Statistical Services,” Journal of the Royal Statistical Society, Series A (General) 145, no. 2 (1982), 195–207, DOI: 10.2307/2981534; and John Kay, “A Better Way to Restore Faith in Official Statistics,” blog post, July 25, 2006, https://www.johnkay.com/2006/07/25/a-better-way-to-restore-faith-in-official-statistics/.

43. Ellen Hughes-Cromwick and Julia Coronado, “The Value of U.S. Government Data to U.S. Business Decisions.”

44. Jackie Mansky, “W. E. B. Du Bois’ Visionary Infographics Come Together for the First Time in Full Color,” Smithsonian, November 15, 2018, https://www.smithsonianmag.com/history/first-time-together-and-color-book-displays-web-du-bois-visionary-infographics-180970826/; and Mona Chalabi, “WEB Du Bois: Retracing His Attempt to Challenge Racism with Data,” Guardian, February 14, 2017, https://www.theguardian.com/world/2017/feb/14/web-du-bois-racism-data-paris-african-americans-jobs.

45. Eric J. Evans, Thatcher and Thatcherism (London: Psychology Press, 2004), 30.

 46. Ian Simpson, Public Confidence in Official Statistics—2016 (London: NatCen Social Research, 2017), https://natcen.ac.uk/media/1361381/natcen_public-confidence-in-official-statistics_web_v2.pdf.

47. The Cabinet Office, Review of Pre-Release Access to Official Statistics, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/62084/pre-release-stats.pdf.

48. Mike Bird, “Lucky, Good or Tipped Off? The Curious Case of Government Data and the Pound,” Wall Street Journal, April 26, 2017; and “New Data Suggest U.K. Government Figures Are Getting Released Early,” Wall Street Journal, March 13, 2017.

Rule Nine: Remember That Misinformation Can Be Beautiful, Too

1. For more information about the life and statistical contribution of Florence Nightingale, see Mark Bostridge, Florence Nightingale: The Woman and Her Legend (New York: Penguin, 2009); Lynn McDonald, ed., The Collected Works of Florence Nightingale (Waterloo, ON: Wilfrid Laurier University Press, 2009–10), and “Florence Nightingale: Passionate Statistician,” Journal of Holistic Nursing 28, no. 1 (March 2010); Hugh Small, “Did Nightingale’s ‘Rose Diagram’ Save Millions of Lives?,” seminar paper, Royal Statistical Society, October 7, 2010; I. Bernard Cohen, “Florence Nightingale,” Scientific American 250, no. 3 (1984), 128–37, JSTOR, www.jstor.org/stable/24969329; Eileen Magnello, “Florence Nightingale: A Victorian Statistician,” Mathematics in School, May 2010, and “The Statistical Thinking and Ideas of Florence Nightingale and Victorian Politicians,” Radical Statistics, issue 102.

2. Draft from John Sutherland (presumed on behalf of Florence Nightingale) to William Farr, March 1861.

3. These quotes from Nightingale are in Marion Diamond and Mervyn Stone, “Nightingale on Quetelet,” Journal of the Royal Statistical Society 1 (1981), 66–79.

4. Alberto Cairo, The Functional Art (San Francisco: Peachpit Press, 2013).

5. Robert Venturi, Denise Scott Brown, and Steven Izenour, Learning from Las Vegas: The Forgotten Symbolism of Architectural Form (Cambridge, MA: MIT Press, 1977); see also Kurt Kohlstedt, “Lessons from Sin City: The Architecture of ‘Ducks’ Versus ‘Decorated Sheds,’” 99% Invisible.org, September 26, 2016, https://99percentinvisible.org/article/lessons-sin-city-architecture-ducks-versus-decorated-sheds/; and Edward Tufte, The Visual Display of Quantitative Information (Cheshire, CT: Graphics Press, 1983, 2001), 106–21.

6. Scott Bateman et al., “Useful Junk? The Effects of Visual Embellishment on Comprehension and Memorability of Charts,” Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI, Atlanta, 2010.

7. Linda Rodriguez McRobbie, “When the British Wanted to Camouflage Their Warships, They Made Them Dazzle,” Smithsonian, April 7, 2016, https://www.smithsonianmag.com/history/when-british-wanted-camouflage-their-warships-they-made-them-dazzle-180958657/.

8. David McCandless, Debtris US, Information Is Beautiful, video, December 30, 2010, https://www.youtube.com/watch?v=K7Pahd2X-eE.

9. David McCandless and Stephanie Starling, “The Billion-Pound-O-Gram,” Information Is Beautiful, data visualization, updated November 2019, https://informationisbeautiful.net/visualizations/the-billion-pound-o-gram.

 10. Brian Brettschneider, “Lessons from Posting a Fake Map,” Forbes.com, November 23, 2018, https://www.forbes.com/sites/brianbrettschneider/2018/11/23/lessons-from-posting-a-fake-map/#5138b31959ec.

11. Florence Nightingale, “Notes on the Health of the British Army,” in McDonald, The Collected Works of Florence Nightingale, vol. 14, 37.

12. See McDonald, The Collected Works of Florence Nightingale, vol. 14, 551–52.

13. Letter from Florence Nightingale to Sidney Herbert, August 19, 1857.

14. Alberto Cairo, How Charts Lie (New York: W. W. Norton, 2019), 47.

15. William Cleveland, The Elements of Graphing Data (Monterey, CA: Wadsworth, 1994); Gene Zelazny, Say It with Charts (New York: McGraw-Hill, 1985); Naomi Robbins, Creating More Effective Graphs (Hoboken, NJ: Wiley, 2005).

16. Edward Tufte, Envisioning Information (Cheshire, CT: Graphics Press, 1990).

17. Larry Buchanan, “Idea of the Week: Inequality and New York’s Subway,” The New Yorker, April 15, 2013, https://www.newyorker.com/news/news-desk/idea-of-the-week-inequality-and-new-yorks-subway.

18. Simon Scarr, “Iraq’s Bloody Toll,” South China Morning Post, December 17, 2011, https://www.scmp.com/infographics/article/1284683/iraqs-bloody-toll.

19. Andy Cotgreave, “Lies, Damned Lies and Statistics,” InfoWorld, June 28, 2016, https://www.infoworld.com/article/3088166/why-how-to-lie-with-statistics-did-us-a-disservice.html.

20. Letter from William Farr to Florence Nightingale, November 24, 1863, quoted in John M. Eyler, Victorian Social Medicine: The Ideas and Methods of William Farr (London: Johns Hopkins University Press, 1979), 175.

21. Science Museum, “Florence Nightingale: The Pioneer Statistician,” December 10, 2018, https://www.sciencemuseum.org.uk/objects-and-stories/florence-nightingale-pioneer-statistician.

Rule Ten: Keep an Open Mind

1. Leon Festinger, Henry Riecken, and Stanley Schachter, When Prophecy Fails (New York: Harper Torchbooks, 1956).

2. Walter A. Friedman, Fortune Tellers: The Story of America’s First Economic Forecasters (Princeton, NJ: Princeton University Press, 2013); Sylvia Nasar, Grand Pursuit (London: Fourth Estate, 2011).

3. Friedman, Fortune Tellers.

4. Irving Fisher, How to Live, 21st ed. (New York: Funk & Wagnalls, 1946).

5. Mark Thornton, The Economics of Prohibition (Salt Lake City: University of Utah Press, 1991).

6. Esther Ingliss-Arkell, “Did a Case of Scientific Misconduct Win the Nobel Prize for Physics?,” Gizmodo, April 22, 2014, https://io9.gizmodo.com/did-a-case-of-scientific-misconduct-win-the-nobel-prize-1565949589.

7. Richard Feynman, “Cargo Cult Science,” transcript of speech at Caltech, 1974, http://calteches.library.caltech.edu/51/2/CargoCult.htm.

8. M. Henrion and B. Fischhoff, “Assessing Uncertainty in Physical Constants,” American Journal of Physics 54 (1986), 791–98, https://doi.org/10.1119/1.14447.

9. Author interview with Jonas Olofsson, January 22, 2020.

10. T. C. Brock and J. L. Balloun, “Behavioral Receptivity to Dissonant Information,” Journal of Personality and Social Psychology 6, no. 4, pt. 1 (1967), 413–28, https://doi.org/10.1037/h0021225.

 11. Baruch Fischhoff and Ruth Beyth, “‘I Knew It Would Happen’: Remembered Probabilities of Once-Future Things,” Organizational Behavior & Human Performance 13, no. 1 (1975), 1–16, https://doi.org/10.1016/0030-5073(75)90002-1.

12. Philip Tetlock, Expert Political Judgment (Princeton, NJ: Princeton University Press, 2005); Philip Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction (New York: Crown, 2015), 184.

13. Welton Chang et al., “Developing Expert Political Judgment: The Impact of Training and Practice on Judgmental Accuracy in Geopolitical Forecasting Tournaments,” Judgment and Decision Making 11, no. 5 (September 2016), 509–26.

14. Philip Tetlock and Dan Gardner, Superforecasting, 127.

15. Sylvia Nasar, Grand Pursuit (London: Fourth Estate, 2011); John Wasik, Keynes’s Way to Wealth (New York: McGraw Hill, 2013).

16. Anne Emberton, “Keynes and the Degas Sale,” History Today 46, no. 1 (January 1996); Jason Zweig, “When Keynes Played Art Buyer,” Wall Street Journal, March 30, 2018; “The Curious Tale of the Economist and the Cezanne [sic] in the Hedge,” BBC News, May 3, 2014, https://www.bbc.co.uk/news/magazine-27226104.

17. David Chambers and Elroy Dimson, “Retrospectives: John Maynard Keynes, Investment Innovator,” Journal of Economic Perspectives 27, no. 3 (2013), 213–28, DOI: 10.1257/jep.27.3.213.

18. M. Deutsch and H. B. Gerard, “A Study of Normative and Informational Social Influences Upon Individual Judgment,” Journal of Abnormal and Social Psychology 51, no. 3 (1955), 629–36, https://doi.org/10.1037/h0046408.

19. Philip Tetlock, “Here’s a robust psychological effect that does not wilt under replication scrutiny,” Twitter, January 6, 2020, 10:09 a.m., https://twitter.com/PTetlock/status/1214202229156016128.

20. Nasar, Grand Pursuit, 314.

21. Friedman, Fortune Tellers.

The Golden Rule: Be Curious

1. Orson Welles, remarks to students at the University of California, Los Angeles, 1941.

2. Onora O’Neill, Reith Lectures 2002: A Question of Trust, Lecture 4: “Trust and Transparency,” http://downloads.bbc.co.uk/rmhttp/radio4/transcripts/20020427_reith.pdf.

3. Dan M. Kahan et al., “‘They Saw a Protest’: Cognitive Illiberalism and the Speech-Conduct Distinction,” February 5, 2011, Cultural Cognition Project Working Paper no. 63; Stanford Law Review 64 (2012); Temple University Legal Studies Research Paper, no. 2011-17, available at https://ssrn.com/abstract=1755706.

4. Dan M. Kahan, “Why Smart People Are Vulnerable to Putting Tribe before Truth,” Scientific American: Observations (blog), December 3, 2018, https://blogs.scientificamerican.com/observations/why-smart-people-are-vulnerable-to-putting-tribe-before-truth/; Brian Resnick, “There May Be an Antidote to Politically Motivated Reasoning. And It’s Wonderfully Simple,” Vox, February 7, 2017, https://www.vox.com/science-and-health/2017/2/1/14392290/partisan-bias-dan-kahan-curiosity; Dan. M. Kahan et al., “Science Curiosity and Political Information Processing,” Political Psychology 38 (2017), 179–99, DOI: 10.1111/pops.12396.

 5. Author interview with Dan Kahan, November 24, 2017.

6. J. Kaplan, S. Gimbel, and S. Harris, “Neural Correlates of Maintaining One’s Political Beliefs in the Face of Counterevidence,” Scientific Reports 6, art. no. 39589 (2016), https://doi.org/10.1038/srep39589.

7. G. Loewenstein, “The Psychology of Curiosity: A Review and Reinterpretation,” Psychological Bulletin 116, no. 1 (1994), 75–98, https://doi.org/10.1037/0033-2909.116.1.75.

8. L. Rozenblit and F. Keil, “The Misunderstood Limits of Folk Science: An Illusion of Explanatory Depth,” Cognitive Science 26 (2002), 521–62, DOI: 10.1207/s15516709cog2605_1.

9. Philip M. Fernbach et al., “Political Extremism Is Supported by an Illusion of Understanding,” Psychological Science 24, no. 6 (2013), 939–46, https://doi.org/10.1177/0956797612464058.

10. Steven Sloman and Philip M. Fernbach, “Asked to Explain, We Become Less Partisan,” New York Times, October 21, 2012.

11. Michael F. Dahlstrom, “Storytelling in Science,” Proceedings of the National Academy of Sciences of the United States of America 111, supp. 4 (2002), 13614–20, DOI: 10.1073/pnas.1320645111.

12. Bruce W. Hardy et al., “Stephen Colbert’s Civics Lesson: How Colbert Super PAC Taught Viewers about Campaign Finance,” Mass Communication and Society 17, no. 3 (2014), 329–53, DOI: 10.1080/15205436.2014.891138.

13. “The Planet Money T-Shirt,” Planet Money podcast, special series, December 11, 2013, https://www.npr.org/series/262481306/planet-money-t-shirt-project-series?t=1580750014093.

14. Economics: The Profession and the Public, seminar held at the Treasury, London, May 5, 2017.

15. “The Cure for Boredom Is Curiosity. There Is No Cure for Curiosity,” Quote Investigator, updated September 26, 2016, https://quoteinvestigator.com/2015/11/01/cure/.

16. “Why is this lying bastard lying to me?” This sentiment courtesy of the renowned British reporter Louis Herren.