Introduction: How to lie with statistics
1 Umberto Eco, Serendipities: Language and Lunacy, London: Hachette, 2015.
2 Robert Matthews, ‘Storks Deliver Babies (p = 0.008)’, Teaching Statistics, 22(2), June 2000, 36–8, http://dx.doi.org/10.1111/1467-9639.00013. Research papers in social science typically say that a relationship is ‘statistically significant’ if p = 0.05, which means that if there was no relationship at all, a pattern at least as clear as the one observed would occur just one in twenty times. The stork paper boasted p = 0.008, which means that if there was in fact no relationship between storks and births, a pattern as clear as the one observed would occur just one in 125 times. The tradition of applying such a statistical significance test is regrettable, for reasons we shall not go into now.
3 Conrad Keating, Smoking Kills, Oxford: Signal Books, 2009, p.xv.
4 Science Museum, Sir Austin Bradford Hill, http://broughttolife.sciencemuseum.org.uk/broughttolife/people/austinhill; Peter Armitage, ‘Obituary: Sir Austin Bradford Hill, 1897–1991’, Journal of the Royal Statistical Society, Series A (Statistics in Society), 154(3), 1991, 482–4, www.jstor.org/stable/2983156
5 Keating, Smoking Kills, pp.85–90.
6 Ibid., p.113.
7 John P.A. Ioannidis, ‘A fiasco in the making?’ Stat, 17 March 2020, https://www.statnews.com/2020/03/17/a-fiasco-in-the-making-as-the-coronavirus-pandemic-takes-hold-we-are-making-decisions-without-reliable-data/
8 ‘Taiwan says WHO failed to act on coronavirus transmission warning’,Financial Times, 20 March 2020, https://www.ft.com/content/2a70a02a-644a-11ea-a6cd-df28cc3c6a68
9 Demetri Sevastopulo and Hannah Kuchler, ‘Donald Trump’s chaotic coronavirus crisis’, Financial Times, 27 March 2020, https://www.ft.com/content/80aa0b58-7010-11ea-9bca-bf503995cd6f
10 David Card, ‘Origins of the Unemployment Rate: The Lasting Legacy of Measurement without Theory’, UC Berkeley and NBER Working Paper, February 2011, http://davidcard.berkeley.edu/papers/origins-of-unemployment.pdf
11 Naomi Oreskes and Eric Conway, Merchants of Doubt, London: Bloomsbury, 2010, Chapter 1; and Robert Proctor, Golden Holocaust, Berkeley and Los Angeles: University of California Press, 2011.
12 Smoking And Health Proposal, Brown and Williamson internal memo, 1969 https://www.industrydocuments.ucsf.edu/tobacco/docs/#id=psdw0147.
13 Kari Edwards and Edward Smith, ‘A Disconfirmation Bias in the Evaluation of Arguments’, Journal of Personality and Social Psychology, 71(1), 1996, 5–24.
14 Oreskes and Conway, Merchants of Doubt.
15 Michael Lewis, ‘Has Anyone Seen the President?’, Bloomberg, 9 February 2018, https://www.bloomberg.com/opinion/articles/2018-02-09/has-anyone-seen-the-president
16 Brendan Nyhan, ‘Why Fears of Fake News Are Overhyped’, Medium, 4 February 2019; and Gillian Tett, ‘The Kids Are Alright: The Truth About Fake News’, Financial Times, 6 February 2019, https://www.ft.com/content/d8f43574-29a1-11e9-a5ab-ff8ef2b976c7?desktop=true&segmentId=7c8f09b9-9b61-4fbb-9430-9208a9e233c8
17 CQ Quarterly: https://library.cqpress.com/cqalmanac/document.php?id=cqal65-1259268; and Alex Reinhart, ‘Huff and Puff’, Significance, 11 (4), 2014.
18 Andrew Gelman, ‘Statistics for Cigarette Sellers’, Chance, 25(3), 2012; Reinhart, ‘Huff and Puff’.
19 How to Lie with Smoking Statistics is stored in the Tobacco Industry Documents library. Alex Reinhart pieced together the manuscript and various documents pertaining to the project: Reinhart, ‘The History of “How To Lie With Smoking Statistics”’, https://www.refsmmat.com/articles/smoking-statistics.html
20 Suzana Herculano-Houzel, ‘What is so special about the human brain?’, talk at TED.com given in 2013: https://www.ted.com/talks/suzana_herculano_houzel_what_is_so_special_about_the_human_brain/transcript?ga_source=embed&ga_medium=embed&ga_campaign=embedT
21 On Galileo’s telescope: https://thonyc.wordpress.com/2012/08/23/refusing-to-look/; and https://www.wired.com/2008/10/how-the-telesco/; and https://thekindlyones.org/2010/10/13/refusing-to-look-through-galileos-telescope/
Rule One: Search your feelings
1 Also known as Star Wars: Episode V; screenplay by Leigh Brackett and Lawrence Kasdan.
2 The van Meegeren case is described in John Godley, The Master Forger, London: Home and Van Thal, 1951; and Van Meegeren: A Case History, London: Nelson, 1967; Noah Charney, The Art of Forgery: The Minds, Motives and Methods of Master Forgers, London: Phaidon, 2015; Frank Wynne, I Was Vermeer, London: Bloomsbury, 2007; the BBC TV programme Fake or Fortune (Series 1, Programme 3, 2011); a series of blog posts by Errol Morris titled ‘Bamboozling Ourselves’ starting on the New York Times website, 20 May 2009; the Boijmans Museum film Van Meegeren’s Fake Vermeers (2010, available on YouTube at https://www.youtube.com/watch?v=NnnkuOz08GQ); and particularly Jonathan Lopez, The Man Who Made Vermeers, London: Houghton Mifflin, 2009.
3 Different accounts exist of exactly how van Meegeren made this confession – another account has van Meegeren equating himself with the Dutch master more directly: ‘The painting in Göring’s hands is not, as you assume, a Vermeer of Delft, but a van Meegeren!’ The quotation in the text is from Frank Wynne’s book I was Vermeer.
4 Ziva Kunda, ‘Motivated Inference: Self-Serving Generation and Evaluation of Causal Theories’, Journal of Personality and Social Psychology, 53(4), 1987, 636–47.
5 Stephen Jay Gould, ‘The median isn’t the message’, Discover 6 June 1985, 40–2.
6 This experiment was described on NPR’s ‘The Hidden Brain’ podcast: You 2.0: The Ostrich Effect, 6 August 2018, https://www.npr.org/templates/transcript/transcript.php?storyId=636133086
7 Nachum Sicherman, George Loewenstein, Duane J. Seppi, Stephen P. Utkus, ‘Financial Attention’, Review of Financial Studies, 29(4), 1 April 2016, 863–97, https://doi.org/10.1093/rfs/hhv073
8 ‘Viral post about someone’s uncle’s coronavirus advice is not all it’s cracked up to be’, Full Fact, 5 March 2020, https://fullfact.org/online/coronavirus-claims-symptoms-viral/
9 Guy Mayraz, ‘Wishful Thinking’, 25 October 2011, http://dx.doi.org/10.2139/ssrn.1955644
10 Linda Babcock and George Loewenstein, ‘Explaining Bargaining Impasse: The Role of Self-Serving Biases’, Journal of Economic Perspectives, 11(1), 1997, 109–26, https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.11.1.109
11 A good summary is Dan Kahan’s blog post, What is Motivated Reasoning? How Does It Work?, http://blogs.discovermagazine.com/intersection/2011/05/05/what-is-motivated-reasoning-how-does-it-workdan-kahan-answers/#.WN5zJ_nyuUm. An excellent survey is Ziva Kunda, ‘The case for motivated reasoning’, Psychological Bulletin, 108(3), 1990, 480–98, http://dx.doi.org/10.1037/0033-2909.108.3.480
12 S. C. Kalichman, L. Eaton, C. Cherry, ‘“There is no proof that HIV causes AIDS”: AIDS denialism beliefs among people living with HIV/AIDS’, Journal of Behavioral Medicine, 33(6), 2010, 432–40, https://doi.org/10.1007/s10865-010-9275-7; and A. B. Hutchinson, E. B. Begley, P. Sullivan, H. A. Clark, B. C. Boyett, S. E. Kellerman, ‘Conspiracy beliefs and trust in information about HIV/AIDS among minority men who have sex with men’, Journal of Acquired Immune Deficiency Syndrome, 45(5), 15 August 2007, 603–5.
13 Tim Harford, ‘Why it’s too tempting to believe the Oxford study’, Financial Times, 27 March 2020, https://www.ft.com/content/14df8908-6f47-11ea-9bca-bf503995cd6f
14 Keith E. Stanovich, Richard F. West and Maggie E. Toplak, ‘Myside Bias, Rational Thinking, and Intelligence’, Current Directions in Psychological Science 22(4), August 2013, 259–64, https://doi.org/10.1177/0963721413480174
15 Charles S. Taber and Milton Lodge, ‘Motivated Skepticism in the Evaluation of Political Beliefs’, American Journal of Political Science, 50(3), July 2006, 755–69, http://www.jstor.org/stable/3694247
16 Kevin Quealy, ‘The More Education Republicans Have, the Less They Tend to Believe in Climate Change’, New York Times, 14 November 2017, https://www.nytimes.com/interactive/2017/11/14/upshot/climate-change-by-education.html
17 Caitlin Drummond and Baruch Fischhoff, ‘Individuals with greater science literacy and education have more polarized beliefs on controversial science topics’, PNAS, 21 August 2017, http://www.pnas.org/content/early/2017/08/15/1704882114
18 Charles Lord, L. Ross and M. R. Lepper, ‘Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence’, Journal of Personality and Social Psychology, 37(11), 1979, 2098–2109.
19 Nicholas Epley and Thomas Gilovich, ‘The Mechanics of Motivated Reasoning’, Journal of Economic Perspectives, 30(3), 2016, 133–40, https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.30.3.133
20 Ari LeVaux, ‘Climate change threatens Montana’s barley farmers – and possibly your beer’, Food and Environment Research Network, 13 December 2017, https://thefern.org/2017/12/climate-change-threatens-montanas-barley-farmers-possibly-beer/
21 Author correspondence with Kris De Meyer, 27 October 2018.
22 Gordon Pennycook, Ziv Epstein, Mohsen Mosleh, Antonio A. Arechar, Dean Eckles and David G. Rand. ‘Understanding and Reducing the Spread of Misinformation Online.’ PsyArXiv. 13 November 2019. https://doi.org/10.31234/osf.io/3n9u8; see also Oliver Burkeman, ‘How to stop the spread of fake news? Pause for a moment’, Guardian, 7 February 2020, https://www.theguardian.com/lifeandstyle/2020/feb/07/how-to-stop-spread-of-fake-news-oliver-burkeman
23 G. Pennycook and D. G. Rand, ‘Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning’, Cognition, 2018, https://doi.org/10.1016/j.cognition.2018.06.011
24 Shane Frederick, ‘Cognitive Reflection and Decision Making’, Journal of Economic Perspectives, 19(4),2005, 25–42, https://doi.org/10.1257/089533005775196732
25 Diane Wolf, Beyond Anne Frank: Hidden Children and Postwar Families in Holland, Berkeley: University of California Press, 2007, Table 1, citing Raul Hilberg, The Destruction of the European Jews (1985).
Rule Two: Ponder your personal experience
1 Muhammad Yunus interviewed by Steven Covey, http://socialbusinesspedia.com/wiki/details/248
2 Transport for London, Travel In London: Report 11, http://content.tfl.gov.uk/travel-in-london-report-11.pdf, figure 10.8, p.202.
3 These numbers were revealed in a freedom of information request – https://www.whatdotheyknow.com/request/journey_demand_and_service_suppl – and they are nicely summarised here: https://www.ianvisits.co.uk/blog/2016/08/05/london-tube-train-capacities/
4 Transport for London, Travel In London: Report 4, http://content.tfl.gov.uk/travel-in-london-report-4.pdf, p.5.
5 Author interview with Lauren Sager Weinstein and Dale Campbell of TfL, 9 July 2019.
6 Ipsos MORI, Perils of Perception 2017, https://www.ipsos.com/ipsos-mori/en-uk/perils-perception-2017
7 ‘“No link between MMR and autism”, major study finds’, NHS News, Tuesday, 5 March 2019, https://www.nhs.uk/news/medication/no-link-between-mmr-and-autism-major-study-finds/
8 ‘When do children usually show symptoms of autism?’, National Institute of Child Health and Clinical Development, https://www.nichd.nih.gov/health/topics/autism/conditioninfo/symptoms-appear
9 David McRaney, ‘You Are Not So Smart Episode 62: Naïve Realism’, https://youarenotsosmart.com/2015/11/09/yanss-062-why-you-often-believe-people-who-see-the-world-differently-are-wrong/; and Tom Gilovich and Lee Ross, The Wisest One in the Room, New York: Free Press, 2016.
10 Ipsos MORI, Perils of Perception 2017, https://www.ipsos.com/ipsos-mori/en-uk/perils-perception-2017
11 David Dranove, Daniel Kessler, Mark McClellan and Mark Satterthwaite, ‘Is More Information Better? The Effects of “Report Cards” on Health Care Providers’, National Bureau of Economic Research Working Paper 8697 (2002), http://www.nber.org/papers/w8697
12 Charles Goodhart, ‘Problems of Monetary Management: The U.K. Experience’, in Anthony S. Courakis (ed.), Inflation, Depression, and Economic Policy in the West, London: Mansell, 1981, pp.111–46. The original paper was presented at a conference in 1975.
13 Donald T. Campbell, ‘Assessing the impact of planned social change’, Evaluation and Program Planning, 2(1), 1979 – an earlier version was published in 1976 and a conference paper existed in 1974.
14 Abhijit Vinayak Banerjee, Dean S. Karlan and Jonathan Zinman, ‘Six randomized evaluations of microcredit: Introduction and further steps’, 2015; and Rachel Meager, ‘Understanding the average effect of microcredit’, https://voxdev.org/topic/methods-measurement/understanding-average-effect-microcredit
15 Anna Rosling Rönnlund, ‘See how the rest of the world lives, organized by income’, TED 2017, anna_rosling_ronnlund_see_how_the_rest_of_the_ world_lives_organized_by_income
Rule Three: Avoid premature enumeration
1 Dr Lucy Smith was interviewed by me and my colleague Richard Fenton-Smith for an episode of More or Less broadcast on BBC Radio 4 on 8 June 2018, https://www.bbc.co.uk/programmes/p069jd0p. The account here is based on our broadcast interview, on discussions over email, and on a phone interview I conducted with Dr Smith on 12 August 2019. Dr Smith’s interviews with people who had lost a baby between twenty and twenty-four weeks of pregnancy are at https://www.healthtalk.org/20-24
2 See Merian F. MacDorman et al, ‘International Comparisons of Infant Mortality and Related Factors: United States and Europe, 2010’, National Vital Statistics Reports, 24 September 2014.
3 Denis Campbell, ‘Concern at rising infant mortality rate in England and Wales’, Guardian, 15 March 2018, https://www.theguardian.com/society/2018/mar/15/concern-at-rising-infant-mortality-rate-in-england-and-wales
4 Peter Davis et al, ‘Rising infant mortality rates in England and Wales – we need to understand gestation specific mortality’, BMJ 361, 8 May 2018, https://doi.org/10.1136/bmj.k1936
5 BBC More or Less, 8 April 2020, https://www.bbc.co.uk/programmes/m000h6cb
6 Author interview with Rebecca Goldin, 12 December 2017.
7 Paul J. C. Adachi and Teena Willoughby, ‘The Effect of Video Game Competition and Violence on Aggressive Behavior: Which Characteristic Has the Greatest Influence?’, Psychology of Violence, 1(4), 2011, 259–74, https://doi.org/10.1037/a0024908
8 ‘Immigration post-Brexit’, Leave Means Leave research paper, http://www.leavemeansleave.eu/research/immigration-post-brexit-fair-flexible-forward-thinking-immigration-policy/
9 Jonathan Portes, ‘Who Are You Calling Low-Skilled?’, UK in a Changing Europe, 12 April 2017, https://ukandeu.ac.uk/who-are-you-calling-low-skilled/
10 Robert Wright, ‘Brexit visa changes to hit sectors in need of low-skilled labour’, Financial Times, 18 February 2020, https://www.ft.com/content/890e84ce-5268-11ea-90ad-25e377c0ee1f
11 https://www.theguardian.com/society/2018/nov/22/concern-over-rise-in-suicide-attempts-among-young-women
12 NHS Digital, Mental Health of Children and Young People in England, 2017, 22 November 2018, https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/2017/2017
13 https://www.nhs.uk/conditions/self-harm/
14 Email correspondence with the NatCen press office, 29 November 2018.
15 Data from official sources such as the Office for National Statistics: https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/suicidesintheunitedkingdom/2017registrations#suicide-patterns-by-age
16 https://www.theguardian.com/business/2014/jan/20/oxfam-85-richest-people-half-of-the-world
17 https://oxfamblogs.org/fp2p/anatomy-of-a-killer-fact-the-worlds-85-richest-people-own-as-much-as-poorest-3-5-billion/; and for the BBC interview with Mr Fuentes see https://www.bbc.com/news/magazine-26613682
18 The underlying data come from the Global Wealth Report, which is published each year by Credit Suisse. The 2013 version supplied the data for Oxfam’s original ‘killer fact’ and it is available online here: https://publications.credit-suisse.com/tasks/render/file/?fileID=BCDB1364-A105-0560-1332EC9100FF5C83
19 ‘Social protection for older persons: Policy trends and statistics 2017–19’, International Labour Office, Social Protection Department, Geneva, 2018; available at https://www.ilo.org/wcmsp5/groups/public/---ed_protect/---soc_sec/documents/publication/wcms_645692.pdf
20 For the UK, the Institute for Fiscal Studies Review of Living Standards, Poverty and Inequality in the UK. For global top incomes, the World Inequality Report. Another good source is Our World In Data. More specific references are provided in the notes below.
Rule Four: Step back and enjoy the view
1 For more reporting on this issue, listen to the 8 June 2018 episode of More or Less, presented by me and researched by my colleagues Richard Fenton-Smith and Richard Vadon: https://www.bbc.co.uk/programmes/p069jd0p
2 Johan Galtung and Mari Holmboe Ruge, ‘The structure of foreign news: The presentation of the Congo, Cuba and Cyprus crises in four Norwegian newspapers’, Journal of Peace Research, 2(1), 1965, 64–90.
3 Max Roser, ‘Stop Saying that 2016 Was the Worst Year’, Washington Post, 29 December 2016, https://www.washingtonpost.com/posteverything/wp/2016/12/29/stop-saying-that-2016-was-the-worst-year/?utm_term=.bad894bad69a; see also NPR’s Planet Money, ‘The Fifty Year Newspaper’, 29 December 2017, https://www.npr.org/templates/transcript/transcript.php?storyId=574662798
4 C. P. Morice, J. J. Kennedy, N. A. Rayner and P. D. Jones, ‘Quantifying uncertainties in global and regional temperature change using an ensemble of observational estimates: The HadCRUT4 dataset’, Journal of Geophysical Research, 117(D8), 2012, https://doi.org/10.1029/2011JD017187, describing data from the Met Office Hadley Centre. The data are charted by and downloadable from ‘Our World in Data’, https://ourworldindata.org/co2-and-other-greenhouse-gas-emissions. In the 1960s, global temperatures were typically around 0.1ºC below the average of 1961–90. In the twenty-first century they’ve typically been about 0.6ºC above that average, and more recently above 0.7ºC. Increase in temperatures, then, over the past fifty years, has been 0.7–0.8ºC.
5 Max Roser, ‘The short history of global living conditions and why it matters that we know it’, 2018, published online at OurWorldInData. org, retrieved from https://ourworldindata.org/a-history-of-globalliving-conditions-in-5-charts; for Child Mortality, Roser cites data from Gapminder and the World Bank.
6 See Figure E4 in the Executive Summary of the 2018 World Inequality Report: https://wir2018.wid.world/files/download/wir2018-summaryenglish.pdf
7 An excellent source is the Institute for Fiscal Studies review of Living Standards, Poverty and Inequality in the UK. I’ve used the 2018 edition, the most recent available at the time of writing: https://www.ifs.org.uk/uploads/R145%20for%20web.pdf
8 A good summary article on inequality around the world is on the Our World in Data website, written by Joe Hasell, an authority on the subject: https://ourworldindata.org/income-inequality-since-1990
9 Author calculations, based on Natsal-3, the third National Survey of Sexual Attitudes and Lifestyles: http://timharford.com/2018/09/is-twitter-more-unequal-than-life-sex-or-happiness/
10 Michael Blastland and Andrew Dilnot, The Tiger That Isn’t, London: Profile Books, 2008.
11 Andrew C. A. Elliott, Is That a Big Number?, Oxford: Oxford University Press, 2018.
12 Tali Sharot, ‘The Optimism Bias’, TED Talk, 2012: https://www.ted.com/talks/tali_sharot_the_optimism_bias/transcript#t-18026
13 Daniel Kahneman, Thinking, Fast and Slow, New York: Farrar, Straus and Giroux, 2010.
14 Ross A. Miller & Karen Albert, ‘If It Leads, It Bleeds (and If It Bleeds, It Leads): Media Coverage and Fatalities in Militarized Interstate Disputes’ Political Communication 2015, 32(1), 61–82, https://doi.org/10.1080/10584609.2014.880976; Barbara Combs & Paul Slovic, ‘Newspaper Coverage of Causes of Death’, Journalism Quarterly, 56(4), 837–43, 849.
15 https://www.cdc.gov/tobacco/data_statistics/fact_sheets/fast_facts/ – there are 1300 deaths a day from smoking-related diseases, about 40,000 a month; almost 3000 people were killed by the 11 September attacks.
16 https://www.ted.com/talks/the_ted_interview_steven_pinker_on_why_our_pessimism_about_the_world_is_wrong/transcript?language=en
17 Steven Pinker mentions in the endnotes of Enlightenment Now (New York: Penguin, 2018), that this correspondence took place in 1982.
18 Quoted in the Guardian, 12 May 2015, https://www.theguardian.com/society/2015/may/12/stroke-association-warns-of-alarming-rise-innumber-of-victims; see also More or Less, 17 May 2015, with the analysis of this claim: https://www.bbc.co.uk/programmes/b05tpz78
19 Oxfam press release, 22 September 2016, http://oxfamapps.org/media/ppdwr
20 A useful survey of various relevant graphs is Max Roser and Mohamed Nagdy, ‘Optimism & Pessimism’, 2018, published online at OurWorldInData.org, retrieved from https://ourworldindata.org/optimism-pessimism – particularly Section I.1 with graphs from Eurobarometer and Ipsos MORI.
21 Martyn Lewis, ‘Not My Idea of Good News’, Independent, 26 April 1993, https://www.independent.co.uk/voices/not-my-idea-of-good-news-at-theend-of-a-week-of-horrifying-events-martyn-lewis-bbc-presenter-argues-1457539.html
22 Max Roser, https://ourworldindata.org/a-history-of-global-livingconditions-in-5-charts – underlying data from the World Bank and from F. Bourguignon and C. Morrisson, ‘Inequality Among World Citizens: 1820–1992’, American Economic Review, 92(4), 2002, 727–48. In 1993 there were 1.94 billion people living in extreme poverty; by 2015 that had fallen to 0.7 billion (705.55 million). The rate of improvement averages 153,600 a day, although of course we have no way of measuring the daily rate as it fluctuates.
23 Samantha Vanderslott, Bernadeta Dadonaite and Max Roser, ‘Vaccination’, 2020. Published online at OurWorldInData.org. Retrieved from: https://ourworldindata.org/vaccination
24 Anna Rosling Rönnlund, Hans Rosling and Ola Rosling, Factfulness, London: Sceptre, 2018.
25 Gillian Tett, ‘Silos and Silences’, Banque de France Financial Stability Review No. 14 – Derivatives – Financial innovation and stability, July 2010, https://core.ac.uk/download/pdf/6612179.pdf
26 Rolf Dobelli, ‘News is bad for you – and giving up reading it will make you happier’, Guardian, 12 April 2013, https://www.theguardian.com/media/2013/apr/12/news-is-bad-rolf-dobelli
27 Nassim Nicholas Taleb, The Bed of Procrustes, London: Penguin Books, 2010.
28 Bill Hanage, Mark Lipsitch, ‘How to Report on the COVID-19 Outbreak Responsibly’, Scientific American, 23 February 2020, https://blogs.scientificamerican.com/observations/how-to-report-on-the-covid-19-outbreak-responsibly/
Rule Five: Get the back story
1 Sheena Iyengar and Mark Lepper, ‘When Choice is Demotivating: Can One Desire Too Much of a Good Thing?’, Journal of Personality and Social Psychology, 79, 2000.
2 Author interview with Benjamin Scheibehenne, October 2009. (I’d like to claim I was ahead of the curve on this one.)
3 B. Scheibehenne, R. Greifeneder and P. M. Todd, ‘Can There Ever Be Too Many Options? A Meta-Analytic Review of Choice Overload’, Journal of Consumer Research, 37, 2010, 409–25, http://scheibehenne.de/ScheibehenneGreifenederTodd2010.pdf
4 ‘Ten Kickstarter Products that Raised the Most Money’: https://www.marketwatch.com/story/10-kickstarter-products-that-raised-the-most-money-2017-06-22-10883052
5 The story is well told in Jordan Ellenberg’s book How Not to Be Wrong (New York: Penguin Press, 2014), with the relevant extract here: https://medium.com/@penguinpress/an-excerpt-from-how-not-to-be-wrong-by-jordan-ellenberg-664e708cfc3d
6 A technical summary (along with some grumbling about how the story has been exaggerated) is in Bill Casselman, ‘The Legend of Abraham Wald’, American Mathematical Society, http://www.ams.org/publicoutreach/feature-column/fc-2016-06
7 An excellent account of the controversy is Daniel Engber, ‘Daryl Bem Proved ESP Is Real Which Means Science Is Broken’, Slate, 17 May 2017, https://slate.com/health-and-science/2017/06/daryl-bem-proved-esp-is-real-showed-science-is-broken.html
8 Chris French, ‘Precognition studies and the curse of the failed replications’, Guardian, 15 March 2012, https://www.theguardian.com/science/2012/mar/15/precognition-studies-curse-failed-replications
9 Nosek was speaking to the Planet Money podcast, episode 677: https://www.npr.org/sections/money/2018/03/07/591213302/episode-677-the-experiment-experiment
10 Brian Nosek has given useful interviews to several podcasts, including You Are Not So Smart (episode 100), https://youarenotsosmart.com/2017/07/19/yanss-100-the-replication-crisis/; Planet Money (episode 677), https://www.npr.org/sections/money/2018/03/07/591213302/episode-677-theexperiment-experiment; EconTalk (16 November 2015), http://www.econtalk.org/brian-nosek-on-the-reproducibility-project/; The Hidden Brain (episode 32), https://www.npr.org/templates/transcript/transcript.php?storyId=477921050; as well as BBC Analysis, ‘The Replication Crisis’, 12 November 2018, https://www.bbc.co.uk/programmes/m00013p9
11 This figure of thirty-nine is based on the subjective opinion of the replicating researchers. Did their results basically back up the original study, or not? That’s a judgement call. An alternative metric is to ask how many of the replication studies produced results that passed the standard (but rather problematic) hurdle of ‘statistical significance’. Only thirty-six did; ninety-seven of the original studies had cleared that hurdle. See ‘Estimating the reproducibility of psychological science’ by the Open Science Collaboration, published in Science, 28 August 2015, 349(6251), https://doi.org/10.1126/science.aac4716.
12 Brief film on YouTube here: https://www.youtube.com/watch?v=n1SJ-Tn3bcQ
13 Planet Money, episode 677: https://www.npr.org/sections/money/2018/03/07/591213302/episode-677-the-experiment-experiment
14 F. J. Anscombe, ‘Fixed-Sample-Size Analysis of Sequential Observations’, Biometrics, 10(1), 1954, 89–100, www.jstor.org/stable/3001665; and Andrew Gelman, Statistical Inference, Modelling and Social Science, blog post 2 May 2018, https://statmodeling.stat.columbia.edu/2018/05/02/continuously-increased-number-animals-statistical-significance-reached-support-conclusions-think-not-bad-actually/
15 David J. Hand, Dark Data, Princeton: Princeton University Press, 2020.
16 Andrew Gelman and Eric Loken, ‘The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time’, working paper, 14 November 2013, http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
17 J. P. Simmons, L. D. Nelson & U. Simonsohn, ‘False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant’, Psychological Science, 22(11), 2011, 1359–66, https://doi.org/10.1177/0956797611417632
18 Kai Kupferschmidt, ‘More and more scientists are preregistering their studies. Should you?’, Science, 21 September 2018.
19 Anjana Ahuja, ‘Scientists strike back against statistical tyranny’, Financial Times, 27 March 2019, https://www.ft.com/content/36f9374c-5075-11e9-8f44-fe4a86c48b33
20 Darrell Huff, How to Lie with Statistics, New York: W. W. Norton, 1993, p.40.
21 John Ioannidis, ‘Why Most Published Research Findings Are False’, PLoS Medicine, 2(8), August 2005, e124, https://doi.org/10.1371/journal.pmed.0020124
22 R. F. Baumeister, E. Bratslavsky, M. Muraven and D. M. Tice, ‘Ego depletion: Is the active self a limited resource?’, Journal of Personality and Social Psychology, 74(5), 1998, 1252–65, http://dx.doi.org/10.1037/0022-3514.74.5.1252; and ‘The End of Ego Depletion Theory?’, Neuroskeptic blog, 31 July 2016, http://blogs.discovermagazine.com/neuroskeptic/2016/07/31/end-of-ego-depletion/#.XGGyflz7SUk
23 Amy Cuddy, ‘Your Body Language May Shape Who You Are’, TED Talk, 2012, https://www.ted.com/talks/amy_cuddy_your_body_language_shapes_who_you_are/transcript?language=en
24 Kahneman, Thinking, Fast and Slow, pp.53–7.
25 Ed Yong, ‘Nobel laureate challenges psychologists to clean up their act’, Nature News, 3 October 2012, https://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535
26 Ben Goldacre, ‘Backwards Step on Looking into the Future’, Guardian, 23 April 2011, https://www.theguardian.com/commentisfree/2011/apr/23/ben-goldacre-bad-science
27 Robin Wrigglesworth, ‘How a herd of cows trampled on human stockpickers’, Financial Times, 21 January 2020, https://www.ft.com/content/563d61dc-3b70-11ea-a01a-bae547046735?
28 Burton Malkiel, ‘Returns from Investing in Equity Funds’, working paper, Princeton University, 1994.
29 Eric Balchunas, ‘How the Vanguard Effect adds up to $1 trillion’, Bloomberg.com, 30 August 2016, https://www.bloomberg.com/opinion/articles/2016-08-30/how-much-has-vanguard-saved-investors-try-1-trillion
30 For an accessible overview, see Ben Goldacre, ‘What doctors don’t know about the drugs they prescribe’, TED Talk, 2012, https://www.ted.com/talks/ben_goldacre_what_doctors_don_t_know_about_the_drugs_they_prescribe/footnotes?language=en
31 Erick Turner et al, ‘Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy’, New England Journal of Medicine, 17 January 2008, https://www.nejm.org/doi/full/10.1056/NEJMsa065779
32 Ben Goldacre, ‘Transparency, Beyond Publication Bias’, talk given to the International Journal of Epidemiology Conference, 2016; available at https://www.badscience.net/2016/10/transparency-beyond-publicationbias-a-video-of-my-super-speedy-talk-at-ije/
33 Ben Goldacre, Henry Drysdale, Aaron Dale, Ioan Milosevic, Eirion Slade, Philip Hartley, Cicely Marston, Anna Powell-Smith, Carl Heneghan and Kamal R. Mahtani, ‘COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time’, Trials, 20(118), 2019, https://doi.org/10.1186/s13063-019-3173-2.
34 Goldacre, ‘Transparency, Beyond Publication Bias’, https://www.badscience.net/2016/10/transparency-beyond-publication-bias-a-video-of-my-super-speedy-talk-at-ije/
35 Amy Sippett, ‘Does the Backfire Effect exist?’, Full Fact, 20 March 2019, https://fullfact.org/blog/2019/mar/does-backfire-effect-exist/; Brendan Nyhan tweet, 20 March 2019, https://twitter.com/BrendanNyhan/status/1108377656414879744
36 Author interview with Richard Thaler, 17 July 2019.
37 BBC Analysis, ‘The Replication Crisis’, 12 November 2018, https://www.bbc.co.uk/programmes/m00013p9
38 Antonio Granado, ‘Slaves to journals, serfs to the web: The use of the internet in newsgathering among European science journalists’, Journalism, 12(7), 2011, 794–813.
39 A. L. Cochrane, ‘Sickness in Salonica: My first, worst, and most successful clinical trial’, British Medical Journal (Clin Res Ed), 289(6460), 1984, 1726–7, https://doi.org/10.1136/bmj.289.6460.1726
40 ‘A Brief History of Cochrane’, https://community.cochrane.org/handbook-sri/chapter-1-introduction/11-cochrane/112-brief-history-cochrane
41 https://www.webmd.com/urinary-incontinence-oab/news/20180522/yoga-may-be-right-move-versus-urinary-incontinence#1
42 https://www.dailymail.co.uk/health/article-2626209/Could-yoga-cure-INCONTINENCE-Exercise-strengthens-pelvic-floor-muscles-reducingleakage.html
43 https://www.hcd.com/incontinence/yoga-incontinence/
44 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4310548/
45 L. S. Wieland, N. Shrestha, Z. S. Lassi, S. Panda, D. Chiaramonte and N. Skoetz, ‘Yoga for treating urinary incontinence in women’, Cochrane Database of Systematic Reviews 2019, 2, Art. No.: CD012668, https://doi.org/10.1002/14651858.CD012668.pub2
1 R. Bond and P. B. Smith, ‘Culture and conformity: A meta-analysis of studies using Asch’s (1952b, 1956) line judgment task’, Psychological Bulletin, 119(1), 1996, 111–37, http://dx.doi.org/10.1037/0033-2909.119.1.111
2 Tim Harford, ‘The Truth About Our Norm-Core’, Financial Times, 12 June 2015, http://timharford.com/2015/06/the-truth-about-our-norm-core/
3 Bond and Smith, ‘Culture and conformity’; and Natalie Frier, Colin Fisher, Cindy Firman and Zachary Bigaouette, ‘The Effects of Group Conformity Based on Sex’, 2016, Celebrating Scholarship & Creativity Day, Paper 83, http://digitalcommons.csbsju.edu/elce_cscday/83
4 Tim Harford, ‘Trump, Brexit and How Politics Loses the Capacity to Shock’, Financial Times, 16 November 2018, https://www.ft.com/content/b730c95c-e82e-11e8-8a85-04b8afea6ea3
5 Caroline Criado Perez, Invisible Women, London: Chatto and Windus, 2019; the interview was broadcast on BBC Radio 4 on 17 May 2019 and is available on the More or Less website: https://www.bbc.co.uk/programmes/m00050rd
6 Peter Hofland, ‘Reversal of Fortune’, Onco’Zine, 30 November 2013, https://oncozine.com/reversal-of-fortune-how-a-vilified-drug-became-alife-saving-agent-in-the-war-against-cancer/
7 R. Dmitrovic, A. R. Kunselman, R. S. Legro, ‘Sildenafil citrate in the treatment of pain in primary dysmenorrhea: a randomized controlled trial’, Human Reproduction, 28(11), November 2013, 2958–65, https://doi.org/10.1093/humrep/det324
8 BBC More or Less, 31 March 2020, https://www.bbc.co.uk/sounds/play/m000h7st
9 Mayra Buvinic and Ruth Levine, ‘Closing the gender data gap’, Significance, 8 April 2016, https://doi.org/10.1111/j.1740-9713.2016.00899.x; and Charlotte McDonald, ‘Is There a Sexist Data Crisis?’, BBC News, 18 May 2016, https://www.bbc.co.uk/news/magazine-36314061
10 Shelly Lundberg, Robert Pollak and Terence J. Wales, ‘Do Husbands and Wives Pool Their Resources? Evidence from the United Kingdom Child Benefit’, 32(3), 1997, 463–80, https://econpapers.repec.org/article/uwpjhriss/v_3a32_3ay_3a1997_3ai_3a3_3ap_3a463-480.htm
11 Buvinic and Levine, ‘Closing the gender data gap’, https://doi.org/10.1111/j.1740-9713.2016.00899.x
12 Suzannah Brecknell, ‘Interview: Full Fact’s Will Moy on lobbyist “nonsense”, official corrections and why we know more about golf than crime stats’, Civil Service World, 5 May 2016, https://www.civilserviceworld.com/articles/interview/interview-full-fact%E2%80%99s-will-moy-lobbyist-%E2%80%9Cnonsense%E2%80%9D-official-corrections-and-why
13 Maurice C. Bryson, ‘The Literary Digest Poll: Making of a Statistical Myth’, American Statistician, 30(4), 1976, 184–5, https://doi.org/10.1080/00031305.1976.10479173; and Peverill Squire, ‘Why the 1936 Literary Digest Poll Failed’, Public Opinion Quarterly, 52(1), 1988, 125–33, www.jstor.org/stable/2749114
14 P. Whiteley, ‘Why Did the Polls Get It Wrong in the 2015 General Election? Evaluating the Inquiry into Pre-Election Polls’, Political Quarterly, 87, 2016, 437–42, https://doi.org/10.1111/1467-923X.12274
15 John Curtice, ‘Revealed: Why the Polls Got It So Wrong in the British General Election’, The Conversation, 14 January 2016, https://theconversation.com/revealed-why-the-polls-got-it-so-wrong-in-thebritish-general-election-53138
16 Nate Cohn, ‘A 2016 Review: Why Key State Polls Were Wrong About Trump’, New York Times, 31 May 2017, https://www.nytimes.com/2017/05/31/upshot/a-2016-review-why-key-state-polls-werewrong-about-trump.html; and Andrew Mercer, Claudia Deane and Kyley McGeeney, ‘Why 2016 election polls missed their mark’, Pew Research Fact Tank blog, 9 November 2015, http://www.pewresearch.org/fact-tank/2016/11/09/why-2016-election-polls-missedtheir-mark/
17 https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/methodologies/2011censusstatisticsforenglandandwalesmarch2011qmi
18 Author interview with Viktor Mayer-Schönberger, March 2014.
19 Pew Research Center Social Media Factsheet, research conducted January 2018, https://www.pewinternet.org/fact-sheet/social-media/
20 Kate Crawford, ‘The Hidden Biases in Big Data’, Harvard Business Review, 1 April 2013, https://hbr.org/2013/04/the-hidden-biases-in-big-data
21 Leon Kelion, ‘Coronavirus: Covid-19 detecting apps face teething problems’, BBC News, 8 April 2020, https://www.bbc.co.uk/news/technology-52215290
22 Kate Crawford, ‘Artificial Intelligence’s White Guy Problem’, New York Times, 25 June 2016, https://www.nytimes.com/2016/06/26/opinion/sunday/artificial-intelligences-white-guy-problem.html
Rule Seven: Demand transparency when the computer says ‘no’
1 Jeremy Ginsberg, Matthew H. Mohebbi, Rajan S. Patel, Lynnette Brammer, Mark S. Smolinski, Larry Brilliant, ‘Detecting influenza epidemics using search engine query data’, Nature, 457 (7232), 19 February 2009, 1012–14, https://doi.org/10.1038/nature07634
2 Parts of this chapter are closely based on my Financial Times magazine article ‘Big Data: Are We Making a Big Mistake?’ (FT, 28 March 2014, https://www.ft.com/content/21a6e7d8-b479-11e3-a09a-00144feabdc0). I interviewed David Hand, Kaiser Fung, Viktor Mayer-Schönberger, David Spiegelhalter and Patrick Wolfe in early 2014 for the piece.
3 David Lazer and Ryan Kennedy, ‘What We Can Learn from the Epic Failure of Google Flu Trends‘, Wired, https://www.wired.com/2015/10/can-learn-epic-failure-google-flu-trends/; and Declan Butler, ‘What Google Flu Got Wrong’, Nature, https://www.nature.com/news/when-google-got-flu-wrong-1.12413
4 https://www.google.org/flutrends/about/
5 D. Lazer, R. Kennedy, G. King and A. Vespignani, ‘The Parable of Google Flu: Traps in Big Data Analysis’, Science 343(6176), March 2014, 1203–5.
6 S. Cook, C. Conrad, A. L. Fowlkes, M. H. Mohebbi, ‘Assessing Google Flu Trends Performance in the United States during the 2009 Influenza Virus A (H1N1) Pandemic’, PLoS ONE 6(8), 2011, e23610, https://doi.org/10.1371/journal.pone.0023610
7 Janelle Shane, You Look Like a Thing and I Love You, New York: Little, Brown, 2019.
8 For comprehensive reporting of this affair, see the Observer/ Guardian website: https://www.theguardian.com/news/series/cambridge-analytica-files
9 Charles Duhigg, ‘How Companies Learn Your Secrets’, New York Times magazine, 19 February 2012, https://www.nytimes.com/2012/02/19/magazine/shopping-habits.html
10 Hannah Fry, Hello World: Being Human in ihe Age of Computers, London: W. W. Norton, 2018.
11 Cathy O’Neil, Weapons of Math Destruction, London: Allen Lane, 2016.
12 Freakonomics radio episode 268: Bad Medicine Pt 1, 16 August 2017, http://freakonomics.com/podcast/bad-medicine-part-1-story-rebroadcast/
13 P. A. Mackowiak, S. S. Wasserman, M. M. Levine, ‘A Critical Appraisal of 98.6°F, the Upper Limit of the Normal Body Temperature, and Other Legacies of Carl Reinhold August Wunderlich’, JAMA, 268(12), 1992, 1578–80, https://doi.org/10.1001/jama.1992.03490120092034
14 Jeffrey Dastin, ‘Amazon scraps secret AI recruiting tool that showed bias against women’, Reuters, 10 October 2018, https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-airecruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
15 Gerd Gigerenzer and Stephanie Kurzenhaeuser, ‘Fast and frugal heuristics in medical decision making’, Science and Medicine in Dialogue: Thinking through particulars and universals, 2005, 3–15.
16 Paul Meehl, Clinical vs. Statistical Prediction, Minneapolis: University of Minnesota Press, 1954.
17 Fry, Hello World.
18 Mandeep K. Dhami and Peter Ayton, ‘Bailing and jailing the fast and frugal way’, Journal of Behavioral Decision Making, 14(2), 2001, https://doi.org/10.1002/bdm.371
19 Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, Sendhil Mullainathan, ‘Human Decisions and Machine Predictions’, Quarterly Journal of Economics, 133(1), February 2018, 237–93, https://doi.org/10.1093/qje/qjx032; see also Cass R. Sunstein, ‘Algorithms, Correcting Biases’, working paper, 12 December 2018.
20 David Jackson and Gary Marx, ‘Data mining program designed to predict child abuse proves unreliable, DCFS says’, Chicago Tribune, 6 December 2017; and Dan Hurley, ‘Can an Algorithm Tell When Kids Are in Danger?’, New York Times magazine, 2 January 2018, https://www.nytimes.com/2018/01/02/magazine/can-an-algorithm-tell-when-kids-are-indanger.html
21 Hurley, ‘Can an Algorithm Tell When Kids Are in Danger?’
22 Andrew Gelman, ‘Flaws in stupid horrible algorithm revealed because it made numerical predictions’, Statistical Modeling, Causal Inference, and Social Science blog, 3 July 2018, https://statmodeling.stat.columbia.edu/2018/07/03/flaws-stupid-horrible-algorithm-revealed-made-numerical-predictions/
23 Sabine Hossenfelder, ‘Blaise Pascal, Florin Périer, and the Puy de Dôme experiment’, http://backreaction.blogspot.com/2007/11/blaise-pascal-florin-p-and-puy-de-d.html; and David Wootton, The Invention of Science: A New History of the Scientific Revolution, London: Allen Lane, 2015, Chapter 8.
24 See, for example, Louis Trenchard More, ‘Boyle as Alchemist’, Journal of the History of Ideas, 2(1), January 1941, 61–76; and ‘The Strange, Secret History of Isaac Newton’s Papers’, a Q&A with Sarah Dry, https://www.wired.com/2014/05/newton-papers-q-and-a/
25 Wootton, The Invention of Science, p.340.
26 James Burke, Connections, Boston: Little, Brown, 1978; reprint with new introduction 1995, p.74.
27 Wootton, The Invention of Science, p.357.
28 https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
29 Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, Aziz Huq, ‘Algorithmic decision making and the cost of fairness’, arXiv:1701.08230; and Sam Corbett-Davies, Emma Pierson, Avi Feller and Sharad Goel, ‘A computer program used for bail and sentencing decisions was labeled biased against blacks. It’s actually not that clear’, Washington Post, 17 October 2016, https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/17/can-an-algorithm-be-racist-our-analysis-is-more-cautious-than-propublicas/
30 Ed Yong, ‘A Popular Algorithm Is No Better at Predicting Crimes than Random People’, The Atlantic, 17 January 2018, https://www.theatlantic.com/technology/archive/2018/01/equivant-compas-algorithm/550646/
31 Ibid.
32 Julia Dressel and Hany Farid, ‘The Accuracy, Fairness and Limits of Predicting Recidivism’, Science Advances 2018, http://advances.sciencemag.org/content/4/1/eaao5580
33 Onora O’Neill’s Reith Lectures on Trust (http://www.bbc.co.uk/radio4/reith2002/) and her TED talk (https://www.ted.com/speakers/onora_o_neill) are both well worth listening to. Themes of intelligent openness are explored in depth in the Royal Society report ‘Science as an Open Enterprise’, 2012, of which O’Neill was an author. In his book The Art of Statistics (London: Penguin, 2019), David Spiegelhalter shows how O’Neill’s principles can be applied to evaluating algorithms.
34 Email interview with Cathy O’Neil, 29 August 2019.
35 Jack Nicas, ‘How YouTube Drives Viewers to the Internet’s Darkest Corners’, Wall Street Journal, 7 February 2018, https://www.wsj.com/articles/how-youtube-drives-viewers-to-the-internets-darkest-corners-1518020478; and Zeynep Tufekci, ‘YouTube, the Great Radicalizer’, New York Times, 10 March 2018, https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-radical.html. But see in contrast Mark Ledwich and Anna Zaitsev, ‘Algorithmic Extremism: Examining YouTube’s Rabbit Hole of Radicalization’, https://arxiv.org/abs/1912.11211
36 Ryan Singal, ‘Netflix spilled your Brokeback Mountain secret, Lawsuit Claims’, Wired, 17 December 2009, https://www.wired.com/2009/12/netflix-privacy-lawsuit/; and Blake Hallinan and Ted Striphas, ‘Recommended for you: the Netflix Prize and the production of algorithmic culture’, New Media and Society, 2016, https://journals.sagepub.com/doi/pdf/10.1177/1461444814538646
Rule Eight: Don’t take statistical bedrock for granted
1 This is a translation of a Danish TV interview, discussed here: https://www.thelocal.se/20150905/hans-rosling-you-cant-trust-the-media
2 Laura Smith, ‘In 1974, a stripper known as the “Tidal Basin Bombshell” took down the most powerful man in Washington’, Timeline, 18 September 2017, https://timeline.com/wilbur-mills-tidal-basin-3c29a8b47ad1; Stephen Green and Margot Hornblower, ‘Mills Admits Being Present During Tidal Basin Scuffle’, Washington Post, 11 October 1974.
3 ‘The Stripper and the Congressman: Fanne Foxe’s Story’, The Rialto Report Podcast, Episode 82, https://www.therialtoreport.com/2018/07/15/fanne-foxe/
4 Alice M. Rivlin, ‘The 40th Anniversary of the Congressional Budget Office’, Brookings: On the Record, 2 March 2015, https://www.brookings.edu/on-the-record/40th-anniversary-of-the-congressional-budget-office/
5 Philip Joyce, ‘The Congressional Budget Office at Middle Age’, Hutchins Center at Brookings, Working Paper #9, 17 February 2015.
6 Quoted in Nancy D. Kates, Starting from Scratch: Alice Rivlin and the Congressional Budget Office, Cambridge: John F. Kennedy School of Government, Harvard University, 1989.
7 Elaine Povich, ‘Alice Rivlin, budget maestro who “helped save Washington” in fiscal crisis, dies at 88’, Washington Post, 14 May 2019, https://www.washingtonpost.com/local/obituaries/alice-rivlin-budget-maestro-who-helped-save-washington-in-fiscal-crisisdies-at-88/2019/05/14/c141c996-0ff9-11e7-ab07-07d9f521f6b5_story.html
8 Andrew Prokop, ‘The Congressional Budget Office, explained’, Vox, 26 June 2017, https://www.vox.com/policy-and-politics/2017/3/13/14860856/congressional-budget-office-cbo-explained
9 John Frendreis and Raymond Tatalovich, ‘Accuracy and Bias in Macroeconomic Forecasting by the Administration, the CBO, and the Federal Reserve Board’, Polity 32(4), 2000, 623–32, accessed 17 January 2020, https://doi.org/10.2307/3235295; Holly Battelle, CBO’s Economic Forecasting Record, Washington DC: Congressional Budget Office, 2010; Committee for a Responsible Federal Budget, ‘Hindsight is 2020: A look back at CBO’s economic forecasting’, January 2013, https://www.crfb.org/blogs/hindsight-2020-look-back-cbos-economic-forecasting
10 Forecast Evaluation Report 2019, Office for Budget Responsibility, December 2019, https://obr.uk/docs/dlm_uploads/Forecast_evaluation_report_December_2019-1.pdf
11 Malcolm Bull, ‘Can the Poor Think?’, London Review of Books, 41(13), 4 July 2019.
12 Bourree Lam, ‘After a Good Jobs Report, Trump Now Believes Economic Data’, The Atlantic, 10 March 2017, https://www.theatlantic.com/business/archive/2017/03/trump-spicer-jobs-report/519273/
13 Esther King, ‘Germany records lowest crime rate since 1992’, Politico, 8 May 2017, https://www.politico.eu/article/germany-crime-rate-lowest-since-1992/
14 For discussion and the full Trump tweets, see Matthew Yglesias, ‘Trump just tweeted that “crime in Germany is way up.” It’s actually at its lowest level since 1992’, Vox, 18 June 2018; and Christopher F. Schuetze and Michael Wolgelenter, ‘Fact Check: Trump’s False and Misleading Claims about Germany’s Crime and Immigration’, New York Times, 18 June 2018.
15 Diane Coyle, GDP: A Brief But Affectionate History, Oxford: Princeton University Press, 2014, pp.3–4.
16 ‘Report on Greek government deficit and debt statistics’, European Commission, 8 January 2010.
17 Beat Balzli, ‘Greek Debt Crisis: How Goldman Sachs Helped Greece to Mask its True Debt’, Der Spiegel, 8 February 2010, https://www.spiegel.de/international/europe/greek-debt-crisis-how-goldman-sachs-helped-greeceto-mask-its-true-debt-a-676634.html
18 The International Statistical Institute has a chronological account of the sorry tale – last updated by G. O’Hanlon and H. Snorrason, July 2018: https://isi-web.org/images/news/2018-07_Court-proceedings-against-Andreas-Georgiou.pdf
19 ‘Commendation of Andreas Georgiou’ – Press Release: International Statistical Association, 18 September 2018, https://www.isi-web.org/images/2018/Press%20release%20Commendation%20for%20Andreas%20Georgiou%20Aug%202018.pdf
20 R. Langkjær-Bain, ‘Trials of a statistician’, Significance, 14, 2017, 14–19, https://doi.org/10.1111/j.1740-9713.2017.01052.x; ‘An Augean Stable’, The Economist, 13 February 2016, https://www.economist.com/theamericas/2016/02/13/an-augean-stable; ‘The Price of Cooking the Books’, The Economist, 25 February 2012, https://www.economist.com/the-americas/2012/02/25/the-price-of-cooking-the-books
21 Langkjær-Bain, ‘Trials of a statistician’.
22 Author interview with Denise Lievesley, 2 July 2018.
23 ‘Tanzania law punishing critics of statistics “deeply concerning”: World Bank’, Reuters, 3 October 2018, https://www.reuters.com/article/us-tanzania-worldbank/tanzania-law-punishing-critics-of-statistics-deeply-concerning-world-bank-idUSKCN1MD17P
24 Amy Kamzin, ‘Dodgy data makes it hard to judge Modi’s job promises’, Financial Times, 8 October 2018, https://www.ft.com/content/1a008ebe-cad4-11e8-9fe5-24ad351828ab
25 Steven Chase and Tavia Grant, ‘Statistics Canada chief falls on sword over census’, Globe and Mail, 21 July 2010, https://www.theglobeandmail.com/news/politics/statistics-canada-chief-falls-on-sword-over-census/article1320915/
26 Langkjær-Bain, ‘Trials of a statistician’.
27 Nicole Acevedo, ‘Puerto Rico faces lawsuits over hurricane death count data’, NBC News, 1 June 2018; and Joshua Barajas, ‘Hurricane Maria’s official death toll is 46 times higher than it was almost a year ago. Here’s why’, PBS Newshour, 30 August 2018, https://www.pbs.org/newshour/nation/hurricane-marias-official-death-toll-is-46-times-higher-than-it-was-almost-a-year-ago-heres-why
28 ‘2011 Census Benefits Evaluation Report’, https://www.ons.gov.uk/census/2011census/2011censusbenefits/2011censusbenefitsevaluationreport#unquantified-benefits; Ian Cope, ‘The Value of Census Statistics’, https://www.ukdataservice.ac.uk/media/455474/cope.pdf
29 Carl Bakker, Valuing the Census, 2014, https://www.stats.govt.nz/assets/Research/Valuing-the-Census/valuing-the-census.pdf
30 Mónica I. Feliú-Mójer, ‘Why Is Puerto Rico Dismantling Its Institute of Statistics?’, Scientific American: Voices, 1 February 2018.
31 https://www.cbo.gov/publication/54965
32 Ellen Hughes-Cromwick and Julia Coronado, ‘The Value of US Government Data to US Business Decisions’, Journal of Economic Perspectives, 33(1), 2019, 131–46, https://doi.org/10.1257/jep.33.1.131.
33 Milton and Rose Friedman, Two Lucky People (1998), quoted in Neil Monnery, ‘Hong Kong’s postwar transformation shows how fewer data can sometimes boost growth’, https://blogs.lse.ac.uk/businessreview/2017/06/30/hong-kongs-postwar-transformation-showshow-fewer-data-can-sometimes-boost-growth/
34 James C. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed, New Haven: Yale University Press, 1998.
35 Perry Link, ‘China: From Famine to Oslo’, New York Review of Books, 13 January 2011.
36 For a discussion of the death toll under Stalin, see Timothy Snyder, ‘Hitler vs. Stalin: Who Killed More?’, New York Review of Books, 10 March 2011 – a more sensitively written piece than the title suggests. For more on the 1937 census, see Daniel Sandford, ‘In Moscow, history is everywhere’, BBC News, 2 November 2012; and Catherine Merridale, ‘The 1937 Census and the Limits of Stalinist Rule’, Historical Journal, 39(1), 1996, and ‘Called to Account’, The Economist, 3 September 2016, https://www.economist.com/finance-and-economics/2016/09/03/called-to-account
37 Merridale, ‘The 1937 Census and the Limits of Stalinist Rule’.
38 Adam Tooze, Statistics and the German State, 1900-1945, Cambridge: Cambridge University Press, 2001, p.257.
39 Author interview with Denise Lievesley, 11 March 2019.
40 Hetan Shah, ‘How to save statistics from the threat of populism’, Financial Times, 21 October 2018, https://www.ft.com/content/ca491f18-d383-11e8-9a3c-5d5eac8f1ab4
41 Nicholas Eberstadt, Ryan Nunn, Diane Whitmore Schanzenbach, Michael R. Strain, ‘“In Order That They Might Rest Their Arguments on Facts”: The Vital Role of Government-Collected Data’, AEI/Hamilton Project report, March 2017.
42 For more on the Rayner Review, see G. Hoinville and T. M. F. Smith, ‘The Rayner Review of Government Statistical Services’, Journal of the Royal Statistical Society, Series A (General) 145(2),1982, 195–207, https://doi.org/10.2307/2981534; and John Kay, ‘A Better Way to Restore Faith in Official Statistics’, 25 July 2006, https://www.johnkay.com/2006/07/25/a-better-way-to-restore-faith-in-official-statistics/
43 Hughes-Cromwick and Coronado, ‘The Value of US Government Data to US Business Decisions’, https://doi.org/10.1257/jep.33.1.131
44 Jackie Mansky, ‘W.E.B. Du Bois’ Visionary Infographics Come Together for the First Time in Full Color’, Smithsonian Magazine, 15 November 2018, https://www.smithsonianmag.com/history/first-time-together-andcolor-book-displays-web-du-bois-visionary-infographics-180970826/; and Mona Chalabi, ‘WEB Du Bois: retracing his attempt to challenge racism with data’, Guardian, 14 February 2017, https://www.theguardian.com/world/2017/feb/14/web-du-bois-racism-data-paris-african-americans-jobs
45 Eric J. Evans, Thatcher and Thatcherism, London: Psychology Press, 2004, p.30.
46 Ian Simpson, Public Confidence in Official Statistics – 2016, London: NatCEN social research, 2017, https://natcen.ac.uk/media/1361381/natcen_public-confidence-in-official-statistics_web_v2.pdf
47 The Cabinet Office, Review of Pre-Release Access to Official Statistics, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/62084/pre-release-stats.pdf
48 Mike Bird, ‘Lucky, Good or Tipped Off? The Curious Case of Government Data and the Pound’, Wall Street Journal, 26 April 2017; and ‘New Data Suggest U.K. Government Figures Are Getting Released Early’, Wall Street Journal, 13 March 2017.
Rule Nine: Remember that misinformation can be beautiful too
1 For more information about the life and statistical contribution of Florence Nightingale, see Mark Bostridge, Florence Nightingale: The Woman and Her Legend, London: Penguin, 2009; Lynn McDonald (ed.), The Collected Works of Florence Nightingale, Waterloo, Ont: Wilfrid Laurier University Press, 2009-10, and ‘Florence Nightingale: Passionate Statistician’, Journal of Holistic Nursing, 28(1), March 2010; Hugh Small, ‘Did Nightingale’s “Rose Diagram” save millions of lives?’, seminar paper, Royal Statistical Society, 7 October 2010; Cohen, I. Bernard. ‘Florence Nightingale’, Scientific American, 250(3), 1984, 128–37, www.jstor.org/stable/24969329, accessed 13 Mar. 2020; Eileen Magnello, ‘Florence Nightingale: A Victorian Statistician’, Mathematics in School, May 2010, and ‘The statistical thinking and ideas of Florence Nightingale and Victorian politicians’, Radical Statistics, 102.
2 Draft from John Sutherland (presumed on behalf of Florence Nightingale) to William Farr, March 1861.
3 These quotes from Nightingale are in Marion Diamond and Mervyn Stone, ‘Nightingale on Quetelet’, Journal of the Royal Statistical Society, 1, 1981, 66–79.
4 Alberto Cairo, The Functional Art, Berkeley, CA: Peachpit Press, 2013.
5 Robert Venturi, Denise Scott Brown, Steven Izenour, Learning from Las Vegas: The Forgotten Symbolism of Architectural Form, Cambridge, MA: MIT Press, 1977; see also https://99percentinvisible.org/article/lessons-sin-cityarchitecture-ducks-versus-decorated-sheds/; and Edward Tufte, The Visual Display of Quantitative Information, Cheshire, CT: Graphics Press, 1983, 2001, pp.106–121.
6 Scott Bateman, Regan L. Mandryk, Carl Gutwin, Aaron Genest, David McDine, Christopher Brooks, ‘Useful Junk? The Effects of Visual Embellishment on Comprehension and Memorability of Charts’, ACM Conference on Human Factors in Computing Systems (CHI), 2010.
7 Linda Rodriguez McRobbie, ‘When the British wanted to camouflage their warships, they made them dazzle’, Smithsonian Magazine, 7 April 2016, https://www.smithsonianmag.com/history/when-british-wanted-camouflage-their-warships-they-made-them-dazzle-180958657/
8 David McCandless, Debtris US, 30 December 2010, https://www.youtube.com/watch?v=K7Pahd2X-eE
9 https://informationisbeautiful.net/visualizations/the-billion-pound-o-gram
10 Brian Brettschneider, ‘Lessons from posting a fake map’, Forbes.com, 23 November 2018, https://www.forbes.com/sites/brianbrettschneider/2018/11/23/lessons-from-posting-a-fake-map/#5138b31959ec
11 Florence Nightingale, ‘Notes on the Health of the British Army’, quoted in Lynn McDonald (ed.), The Collected Works of Florence Nightingale, vol. 14, p.37.
12 McDonald (ed.), The Collected Works of Florence Nightingale, vol. 14, p.551.
13 Letter from Florence Nightingale to Sidney Herbert, 19 August 1857.
14 Alberto Cairo, How Charts Lie, New York: W. W. Norton, 2019, p.47.
15 William Cleveland, The Elements of Graphing Data, Wadsworth: Monterey, 1994; Gene Zelazny, Say it with Charts, New York: McGraw-Hill, 1985; Naomi Robbins, Creating More Effective Graphs, New Jersey: Wiley, 2005.
16 Edward Tufte, Envisioning Information, Cheshire CT: Graphics Press, 1990.
17 Larry Buchanan, ‘Idea of the Week: Inequality and New York’s Subway’, New Yorker, 15 April 2013, https://www.newyorker.com/news/news-desk/idea-of-the-week-inequality-and-new-yorks-subway
18 Simon Scarr, ‘Iraq’s Bloody Toll’, South China Morning Post, https://www.scmp.com/infographics/article/1284683/iraqs-bloody-toll
19 Andy Cotgreave, ‘Lies, Damned Lies and Statistics’, InfoWorld, https://www.infoworld.com/article/3088166/why-how-to-lie-with-statistics-did-us-a-disservice.html
20 Letter from William Farr to Florence Nightingale, 24 November 1863, quoted in John M. Eyler, Victorian Social Medicine: The Ideas and Methods of William Farr, London: Johns Hopkins Press, 1979, p.175.
21 https://www.sciencemuseum.org.uk/objects-and-stories/florence-nightingale-pioneer-statistician
Rule Ten: Keep an open mind
1 Leon Festinger, Henry Riecken and Stanley Schachter, When Prophecy Fails, New York: Harper-Torchbooks, 1956.
2 Walter A. Friedman, Fortune Tellers: The Story of America’s First Economic Forecasters, Princeton: Princeton University Press, 2013; and Sylvia Nasar, Grand Pursuit, London: Fourth Estate, 2011.
3 Friedman, Fortune Tellers.
4 Irving Fisher, How to Live, New York: Funk and Wagnalls, 21st edition, 1946.
5 Mark Thornton, The Economics of Prohibition, Salt Lake City: University of Utah Press, 1991.
6 Esther Ingliss-Arkell, ‘Did a case of scientific misconduct win the Nobel prize for physics?’, https://io9.gizmodo.com/did-a-case-of-scientific-misconduct-win-the-nobel-prize-1565949589
7 Richard Feynman, ‘Cargo Cult Science’, speech at Caltech, 1974: http://calteches.library.caltech.edu/51/2/CargoCult.htm
8 M. Henrion and B. Fischhoff, ‘Assessing Uncertainty in Physical Constants’, American Journal of Physics, 54, 1986, 791–8, https://doi.org/10.1119/1.14447
9 Author interview with Jonas Olofsson, 22 January 2020.
10 T. C. Brock and J. L. Balloun, ‘Behavioral receptivity to dissonant information’, Journal of Personality and Social Psychology, 6(4, Pt.1), 1967, 413–28, https://doi.org/10.1037/h0021225
11 B. Fischhoff and R. Beyth, ‘“I knew it would happen”: Remembered probabilities of once-future things’, Organizational Behavior & Human Performance, 13(1), 1975, 1–16, https://doi.org/10.1016/0030-5073(75)90002-1
12 Philip Tetlock, Expert Political Judgement, Princeton: Princeton University Press, 2005; Philip Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction, New York: Crown, 2015, p.184.
13 Welton Chang, Eva Chen, Barbara Mellers, Philip Tetlock, ‘Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments’, Judgment and Decision Making, 11(5), September 2016, 509–26.
14 Tetlock and Gardner, Superforecasting, p.127.
15 Nasar, Grand Pursuit; and John Wasik, Keynes’s Way to Wealth, New York: McGraw-Hill, 2013.
16 Anne Emberton, ‘Keynes and the Degas Sale’, History Today, 46(1), January 1996; Jason Zweig, ‘When Keynes Played Art Buyer’, Wall Street Journal, 30 March 2018; ‘The Curious Tale of the Economist and the Cezanne in the Hedge’, 3 May 2014, https://www.bbc.co.uk/news/magazine-27226104
17 David Chambers and Elroy Dimson, ‘Retrospectives: John Maynard Keynes, Investment Innovator’, Journal of Economic Perspectives, 27(3), 2013, 213-28, https://doi.org/10.1257/jep.27.3.213
18 M. Deutsch and H. B. Gerard, ‘A study of normative and informational social influences upon individual judgment’, Journal of Abnormal and Social Psychology, 51(3), 1955, 629–36, https://doi.org/10.1037/h0046408
19 Philip Tetlock, Twitter, 6 January 2020, https://twitter.com/PTetlock/status/1214202229156016128.
20 Nasar, Grand Pursuit, p.314.
21 Friedman, Fortune Tellers.
The Golden Rule: Be curious
1 Orson Welles, remarks to students at the University of California Los Angeles, 1941.
2 Onora O’Neill, Reith Lectures 2002, Lecture 4: ‘Trust and transparency’, http://downloads.bbc.co.uk/rmhttp://radio4/transcripts/20020427_reith.pdf
3 Dan M. Kahan, David A. Hoffman, Donald Braman, Danieli Evans Peterman and Jeffrey John Rachlinski, ‘“They Saw a Protest”: Cognitive Illiberalism and the Speech-Conduct Distinction’, 5 February 2011, Cultural Cognition Project Working Paper no. 63; Stanford Law Review, 64, 2012; Temple University Legal Studies Research Paper no. 2011–17, available at: https://ssrn.com/abstract=1755706
4 Dan Kahan, ‘Why Smart People Are Vulnerable to Putting Tribe Before Truth’, Scientific American: Observations, 3 December 2018, https://blogs.scientificamerican.com/observations/why-smart-people-are-vulnerable-to-putting-tribe-before-truth/; Brian Resnick, ‘There may be an antidote to politically motivated reasoning. And it’s wonderfully simple’, Vox.com, 7 February 2017, https://www.vox.com/science-andhealth/2017/2/1/14392290/partisan-bias-dan-kahan-curiosity; D. M. Kahan, A. Landrum, K. Carpenter, L. Helft and K. Hall Jamieson, ‘Science Curiosity and Political Information Processing’, Political Psychology, 38, 2017, 179–99, https://doi.org/10.1111/pops.12396
5 Author interview with Dan Kahan, 24 November 2017.
6 J. Kaplan, S. Gimbel and S. Harris, ‘Neural correlates of maintaining one’s political beliefs in the face of counterevidence’, Scientific Reports, 6(39589), 2016, https://doi.org/10.1038/srep39589
7 G. Loewenstein, ‘The psychology of curiosity: A review and reinterpretation’, Psychological Bulletin, 116(1), 1994, 75–98, https://doi.org/10.1037/0033-2909.116.1.75
8 L. Rozenblit and F. Keil, ‘The misunderstood limits of folk science: an illusion of explanatory depth’, Cognitive Science, 26, 2002, 521–62, https://doi.org/ 10.1207/s15516709cog2605_1
9 P. M. Fernbach, T. Rogers, C. R. Fox and S. A. Sloman, ‘Political Extremism Is Supported by an Illusion of Understanding’, Psychological Science, 24(6), 2013, 939–46, https://doi.org/10.1177/0956797612464058
10 Steven Sloman and Philip M. Fernbach, ‘Asked to explain, we become less partisan’, New York Times, 21 October 2012.
11 Michael F. Dahlstrom, ‘Storytelling in science’, Proceedings of the National Academy of Sciences, 111 (Supplement 4), September 2014, 13614–20, https://doi.org/10.1073/pnas.1320645111
12 Bruce W. Hardy, Jeffrey A. Gottfried, Kenneth M. Winneg and Kathleen Hall Jamieson, ‘Stephen Colbert’s Civics Lesson: How Colbert Super PAC Taught Viewers About Campaign Finance, Mass Communication and Society’, Mass Communication and Society 17(3), 2014, 329–53, https://doi.org/10.1080/15205436.2014.891138
13 ‘The Planet Money T-Shirt’: https://www.npr.org/series/262481306/planet-money-t-shirt-project-series?t=1580750014093
14 Economics: The Profession and the Public, seminar held at the Treasury in London, 5 May 2017.
15 Quote Investigator: https://quoteinvestigator.com/2015/11/01/cure/
16 ‘Why is this lying bastard lying to me?’ – this sentiment courtesy of the renowned British reporter Louis Herren.