INTRODUCTION
How to Lie with Statistics
The truly genuine problem . . . does not consist of proving something false but in proving that the authentic object is authentic.
• Umberto Eco1
You know the old story about storks delivering babies? It’s true. I can prove it with statistics. Take a look at the estimated population of storks in each country, and then at the number of babies born each year. Across Europe, there’s a remarkably strong relationship. More storks, more babies; fewer storks, fewer babies.
The pattern is easily strong enough to pass a traditional hurdle for publication in an academic journal. In fact, a scientific paper has been published with the title “Storks Deliver Babies (p = 0.008).” Without getting too technical, all those zeros tell us that this is not a coincidence.2
Perhaps you have already guessed the trick. Large European countries such as Germany, Poland, and Turkey are home to many babies and many storks. Small countries such as Albania and Denmark have few babies and few storks. While there’s a clear pattern in the data, that pattern does not mean that storks cause babies to appear.
You can “prove” anything with statistics, it seems—even that storks deliver babies.
You’d certainly have gotten that impression from reading How to Lie with Statistics. Published in 1954 by a little-known American freelance journalist named Darrell Huff, this wisecracking, cynical little book immediately received a rave review from the New York Times and went on to become perhaps the most popular book on statistics ever published, selling well over a million copies.
The book deserves the popularity and the praise. It’s a marvel of statistical communication. It also made Darrell Huff a nerd legend. Ben Goldacre, an epidemiologist and bestselling author of Bad Science, has written admiringly of how “The Huff” had written a “ripper.” The American writer Charles Wheelan describes his book Naked Statistics as “an homage” to Huff’s “classic.” The respected journal Statistical Science organized a Huff retrospective fifty years after its publication.
I used to feel the same way. As a teenager, I loved reading How to Lie with Statistics. Bright, sharp, and illustrated throughout with playful cartoons, the book gave me a peek behind the curtain of statistical manipulation, showing me how the swindling was done so that I would not be fooled again.
Huff is full of examples. He begins by pondering how much money Yale graduates make. According to a 1950 survey, the class of 1924 had an average income of close to $500,000 a year in today’s terms. That is just plausible enough to believe—this is Yale, after all—but half a million dollars a year is a lot of money. Is that really the average?
No. Huff explains that this “improbably salubrious” figure comes from self-reported data, which means we can expect people to exaggerate their income for the sake of vanity. Furthermore, the survey is only of people who bothered to respond—and only those alumni Yale could find. And who are easily found? The rich and famous. “Who are the little lost sheep down in the Yale rolls as ‘address unknown’?” asks Huff. Yale will keep track of the millionaire alumni, but some of the also-ran graduates might easily have slipped through the net. All this means that the survey will present a grossly inflated view.
Huff briskly moves on through a vast range of statistical crimes, from toothpaste advertisements based on cherry-picked research to maps that change their meaning depending on how you color them in. As Huff wrote, “The crooks already know these tricks; honest men must learn them in self-defense.”
If you read How to Lie with Statistics, you will come away more skeptical about the ways numbers can deceive you. It’s a clever and instructive book.
But I’ve spent more than a decade trying to communicate statistical ideas and fact-check numerical claims—and over the years, I’ve become more and more uneasy about How to Lie with Statistics and what that little book represents. What does it say about statistics—and about us—that the most successful book on the subject is, from cover to cover, a warning about misinformation?
Darrell Huff published How to Lie with Statistics in 1954. But something else happened that very same year: two British researchers, Richard Doll and Austin Bradford Hill, produced one of the first convincing studies to demonstrate that smoking cigarettes causes lung cancer.3
Doll and Hill could not have figured this out without statistics. Lung cancer rates had increased sixfold in the UK in just fifteen years; by 1950 the UK had the highest in the world, and deaths from lung cancer exceeded deaths from tuberculosis for the first time. Even to realize that this was happening required a statistical perspective. No single doctor would have formed more than an anecdotal impression.
As for showing that cigarettes were to blame—again, statistics were essential. A lot of people thought that motorcars were the cause of the rise in lung cancer. That made perfect sense. In the first half of the twentieth century, motorcars became commonplace, with their exhaust fumes and the strangely compelling vapor from the tar in new roads. Lung cancer increased at the same time. Figuring out the truth—that it was cigarettes rather than cars that caused lung cancer—required more than simply looking around. It required researchers to start counting, and comparing, with care. More concisely, it required statistics.
The cigarette hypothesis was viewed with skepticism by many, although it was not entirely new. For example, there had been a big research effort in Nazi Germany to produce evidence that cigarettes were dangerous; Adolf Hitler despised smoking. The Führer was no doubt pleased when German doctors discovered that cigarettes caused cancer. For obvious reasons, though, “hated by Nazis” was no impediment to the popularity of tobacco.
So Doll and Hill decided to conduct their own statistical investigations. Richard Doll was a handsome, quiet, and unfailingly polite young man. He had returned from the Second World War with a head full of ideas about how statistics could revolutionize medicine. His mentor, Austin Bradford Hill, had been a pilot in the First World War before nearly dying of tuberculosis.* Hill was a charismatic man, had a sharp wit, and was said to be the finest medical statistician of the twentieth century.4 Their work together as data detectives was to prove lifesaving.
The pair’s first smoking-and-cancer study began on New Year’s Day 1948. It was centered around twenty hospitals in northwest London, and Richard Doll was in charge. Every time a patient arrived in a hospital with cancer, nurses would—at random—find someone else in the same hospital of the same sex and about the same age. Both the cancer patients and their counterparts would be quizzed in depth about where they lived and worked, their lifestyle and diet, and their history of smoking. Week after week, month after month, the results trickled in.
In October 1949, less than two years after the trial began, Doll stopped smoking. He was thirty-seven, and had been a smoker his entire adult life. He and Hill had discovered that heavy smoking of cigarettes didn’t just double the risk of lung cancer, or triple the risk, or even quadruple the risk. It made you sixteen times more likely to get lung cancer.5
Hill and Doll published their results in September 1950, and promptly embarked on a bigger, longer-term, and more ambitious trial. Hill wrote to every doctor in the UK—all 59,600 of them—asking them to complete a “questionary” about their health and smoking habits. Doll and Hill figured that the doctors would be capable of keeping track of what they smoked. They would stay on the medical register, so they’d always be easy to find. And when a doctor dies, you can expect a good diagnosis as to the cause of death. All Hill and Doll had to do was wait.
More than 40,000 doctors responded to Hill’s request, but not all of them were delighted. You have to understand that smoking was extremely common at the time, and it was no surprise to find that 85 percent of the male doctors in Doll and Hill’s initial sample were smokers. Nobody likes to be told that they might be slowly killing themselves, especially if the suicide method is highly addictive.
One doctor buttonholed Hill at a London party. “You’re the chap who wants us to stop smoking,” he pointedly declared.
“Not at all,” replied Hill, who was still a pipe smoker himself. “I’m interested if you go on smoking to see how you die. I’m interested if you stop because I want to see how you die. So you choose for yourself, stop or go on. It’s a matter of indifference to me. I shall score up your death anyway.”6
Did I mention that Hill originally trained as an economist? It’s where he learned his charm.
The study of doctors rolled on for decades, but it wasn’t long before Doll and Hill had enough data to publish a clear conclusion: Smoking causes lung cancer, and the more you smoke the higher the risk. What’s more—and this was new—smoking causes heart attacks, too.
Doctors aren’t fools. In 1954, when the research was published in the doctors’ own professional magazine, the British Medical Journal, they could draw their own conclusions. Hill quit smoking that year, and many of his fellow doctors quit, too. Doctors became the first identifiable social group in the UK to give up smoking in large numbers.
In 1954, then, two visions of statistics had emerged at the same time. To the many readers of Darrell Huff’s How to Lie with Statistics, statistics were a game, full of swindlers and cheats—and it could be amusing to catch the scoundrels at their tricks. But for Austin Bradford Hill and Richard Doll, statistics were no laughing matter. Their game had the highest imaginable stakes, and if it was played honestly and well, it would save lives.
By the spring of 2020—as I was putting the finishing touches to this book—the high stakes involved in rigorous, timely, and honest statistics had suddenly become all too clear. A new coronavirus was sweeping the world. Politicians had to make the most consequential decisions in decades, and fast. Many of those decisions depended on data detective work that epidemiologists, medical statisticians, and economists were scrambling to conduct. Tens of millions of lives were potentially at risk. So were billions of people’s livelihoods.
As I write these words, it is early April 2020: countries around the world are a couple of weeks into lockdowns, global deaths have just passed 60,000, and it is far from clear how the story will unfold. Perhaps, by the time this book is in your hands, we will be mired in the deepest economic depression since the 1930s and the death toll will have mushroomed. Perhaps, by human ingenuity or good fortune, such apocalyptic fears will have faded into memory. Many scenarios seem plausible. And that’s the problem.
An epidemiologist, John Ioannidis, wrote in mid-March that COVID-19 may be “a once-in-a-century evidence fiasco.”7 The data detectives are doing their best—but they’re having to work with data that are patchy, inconsistent, and woefully inadequate for making life-and-death decisions with the confidence we’d like.
Details of the fiasco will, no doubt, be studied for years to come. But some things already seem clear. At the beginning of the crisis, for example, politics seem to have impeded the free flow of honest statistics—a problem we’ll return to in the eighth chapter. Taiwan has complained that in late December 2019 it had given important clues about human-to-human transmission to the World Health Organization—but as late as mid-January, the WHO was reassuringly tweeting that China had found no evidence of human-to-human transmission. (Taiwan is not a member of the WHO, because China claims sovereignty over the territory and demands that it should not be treated as an independent state. It’s possible that this geopolitical obstacle led to the alleged delay.)8
Did this matter? Almost certainly; with cases doubling every two or three days, we will never know what might have been different with an extra couple of weeks of warning. It’s clear that many leaders took their time before appreciating the potential gravity of the threat. President Trump, for instance, announced in late February, “It’s going to disappear. One day, it’s like a miracle, it will disappear.” Four weeks later, with 1,300 Americans dead and more confirmed cases in the United States than any other country, Mr. Trump was still talking hopefully about getting everybody to church at Easter.9
As I write, debates are raging. Can rapid testing, isolation, and contact tracing contain outbreaks indefinitely, or only delay their spread? Should we worry more about small indoor gatherings or large outdoor gatherings? Does closing schools help prevent the spread of the virus, or do more harm as children go to stay with vulnerable grandparents? How much does wearing masks help? These and many other questions can be answered only by good data on who has been infected, and when.
But a vast number of infections were not being registered in official statistics, due to a lack of tests—and the tests that were being conducted were giving a skewed picture, being focused on medical staff, critically ill patients, and—let’s face it—rich, famous people. At the time of writing, the data simply can’t yet tell us how many mild or asymptomatic cases there are—and hence how deadly the virus really is. As the death toll rose exponentially in March—doubling every two days—there was no time to wait and see. Leaders put economies into an induced coma—more than three million Americans filed jobless claims in a single week in late March, five times the previous record. The following week was even worse: another six and a half million claims were filed. Were the potential health consequences really catastrophic enough to justify sweeping away so many people’s incomes? It seemed so—but epidemiologists could only make their best guesses with very limited information.
It’s hard to imagine a more extraordinary illustration of how much we usually take accurate, systematically gathered numbers for granted. The statistics for a huge range of important issues that predate the coronavirus have been painstakingly assembled over the years by diligent statisticians, and often made available to download, free of charge, anywhere in the world. Yet we are spoiled by such luxury, casually dismissing “lies, damned lies, and statistics.” The case of COVID-19 reminds us how desperate the situation can become when the statistics simply aren’t there.
Darrell Huff made statistics seem like a stage magician’s trick: all good fun but never to be taken seriously. Long before the coronavirus, I’d started to worry that this isn’t an attitude that helps us today. We’ve lost our sense that statistics might help us make the world add up. It’s not that we feel every statistic is a lie, but that we feel helpless to pick out the truths. So we believe whatever we want to believe (more on that in the next chapter), and for the rest we adopt Huff’s response: a harsh laugh, a shrug, or both.
This statistical cynicism is not just a shame—it’s a tragedy. If we give in to a sense that we no longer have the power to figure out what’s true, then we’ve abandoned a vital tool. It’s a tool that showed us that cigarettes are deadly. It’s our only real chance of finding a way through the coronavirus crisis—or, more broadly, understanding the complex world in which we live. But the tool is useless if we lapse into a reflexive dismissal of any unwelcome statistical claim. Of course, we shouldn’t be credulous—but the antidote to credulity isn’t to believe nothing, but to have the confidence to assess information with curiosity and a healthy skepticism.
Good statistics are not a trick, although they are a kind of magic. Good statistics are not smoke and mirrors; in fact, they help us see more clearly. Good statistics are like a telescope for an astronomer, a microscope for a bacteriologist, or an X-ray for a radiologist. If we are willing to let them, good statistics help us see things about the world around us and about ourselves—both large and small—that we would not be able to see in any other way.
My main aim with this book is to persuade you to embrace Doll and Hill’s vision, not Huff’s cynicism. I want to convince you that statistics can be used to illuminate reality with clarity and honesty. To do that, I need to show you that you can use statistical reasoning for yourself, sizing up the claims that surround you in the media, on social media, and in everyday conversation. I want to help you evaluate statistics from scratch, and just as important, to figure out where to find help that you can trust.
The good news is that this is going to be fun. There’s a real satisfaction in getting to the bottom of the statistical story. You gain in confidence and feed your curiosity along the way, and end up feeling that you’ve mastered something. You understand rather than sneer from the sidelines. Darrell Huff’s approach is junk food: superficially appealing but tedious after a while. And it’s bad for you. But the opposite of statistical junk food isn’t raw oats and turnips; it’s a satisfying and delightfully varied menu.
In this book I’ll be describing what I’ve learned myself since 2007, when the BBC asked me to present a radio program called More or Less, a show about numbers in the news and in life. The show’s creators, the journalist Michael Blastland and the economist Andrew Dilnot, were moving on. I was less well qualified for the role than the BBC might have imagined: I trained in economic theory, not statistics. Yes, that training gave me some self-assurance when it came to numbers, but it was mostly defensive. I’d learned to spot flaws and tricks, but couldn’t do much beyond that.
It was there that my journey away from the viewpoint of Darrell Huff began.
Week after week, my colleagues and I would evaluate the statistical claims that had emerged out of the mouths of politicians or been printed in large type in the newspapers. Those claims often stretched the truth, but by itself a simple fact-check never seemed like a satisfying response. We would find that behind each claim—true, false, or borderline—was a fascinating world to explore and explain. Whether we were evaluating the prevalence of strokes, the evidence that debt damages economic growth, or even the number of times in The Hobbit that the word “she” is used, the numbers could illuminate the world as well as obscure it.
As the coronavirus pandemic has so starkly illustrated, we depend on reliable numbers to shape our decisions—as individuals, as organizations, and as a society. And just as with coronavirus, the statistics have often been gathered only when we’ve been faced with a crisis. Consider the unemployment rate—a measure of how many people want jobs but do not have them. It’s now a basic piece of information for any government wanting to understand the state of the economy, but back in 1920, nobody could have told you how many people were searching for work.10 Only when severe recessions made the question more politically pertinent did governments begin to collect the data that could answer it.
Our big, bewildering world is full of questions that only careful attention to the numbers can answer. Does Facebook tend to make us happy or sad, and can we predict why different people react in different ways? How many species are in danger of extinction, is that a big proportion of the total, and is the cause climate change, the spread of human agriculture, or something else entirely? Is human innovation speeding up, or slowing down? How serious is the impact of the opioid crisis on the health of middle America? Is teenage drinking becoming less common—and if so, why?
I grew increasingly uneasy when fans of More or Less complimented the way we “debunked false statistics.” Sure, we did that, and it was fun. But slowly, learning on the job, I came to appreciate that the real joy was not in shooting down falsehoods but in trying to understand what was true.
Working on More or Less, I learned that commonsense principles can get you a surprisingly long way as a data detective. It’s these principles I shall sum up in this book. Most of the team of researchers and producers, like me, lacked any serious training in how to handle numbers. But even in highly technical areas, some simple questions—and perhaps an internet search or two—would often produce very rewarding answers. Yes, sometimes an advanced degree in statistics would have been useful, but we never needed it to ask the right questions. You don’t need it either.
Just before Christmas in 1953, senior tobacco executives met at the Plaza Hotel in New York. Doll and Hill’s big study wouldn’t be published until the following year, but the cigarette companies were already aware that the science was starting to look pretty bad for them. They met to figure out how to respond to this looming crisis.
Their answer was—alas—quite brilliant, and set the standard for propaganda ever since.
They muddied the waters. They questioned the existing research; they called for more research; they funded research into other things they might persuade the media to get excited about, such as sick building syndrome or mad cow disease. They manufactured doubt.11 A secret industry memo later reminded insiders that “doubt is our product.”12
Understandably, when we think about persuasion, we think about people being tricked into believing something that they shouldn’t—and we’ll discuss this problem in the next chapter. But sometimes the problem is not that we are too eager to believe something, but that we find reasons not to believe anything. Smokers liked smoking, were physically dependent on nicotine, and wanted to keep smoking if they could. A situation where smokers shrugged and said to themselves, “I can’t figure out all these confusing claims,” was a situation that suited the tobacco industry well. Their challenge was not to convince smokers that cigarettes were safe, but to create doubt about the statistical evidence that showed they were dangerous.
And it turns out that doubt is a really easy product to make. A couple of decades ago, psychologists Kari Edwards and Edward Smith conducted an experiment in which they asked people in the United States to produce arguments in favor of and against the politically fraught positions of the day such as abortion rights, spanking children, allowing homosexual couples to adopt, quotas for hiring minorities, and the death penalty for those under sixteen years of age.13 Unsurprisingly, they found that people had biases: the study participants found it hard to construct the kind of arguments that their opponents would use to defend their views. More strikingly, Edwards and Smith showed that those biases tended to appear more clearly in negative arguments. Disbelief flowed more fluidly than belief. The experimental subjects found it much easier to argue against positions they disliked than in favor of those they supported. There was a special power in doubt.
Doubt is also easy to sell because it is a part of the process of scientific exploration and debate. Most of us are—or should be—taught at school to question the evidence. The motto of one of the oldest scientific societies, the Royal Society, is Nullius in verba—“Take nobody’s word for it.” A lobby group seeking to deny the statistical evidence will always be able to point to some aspect of the current science that is not settled, note that the matter is terribly complicated, and call for more research. And these claims will sound scientific, even rather wise. Yet they give a false and dangerous impression: that nobody really knows anything.
The techniques of the tobacco industry have been widely embraced.14 They are used today most obviously by climate change deniers, but they have spread beyond scientific questions and into politics. Robert Proctor, a historian who has spent decades studying the tobacco industry, calls modern politics “a golden age of ignorance.” Much as many smokers would like to keep smoking, many of us are fondly attached to our gut instincts on political questions. All politicians need to do is persuade us to doubt evidence that would challenge those instincts.
As Donald Trump’s former right-hand man Steve Bannon infamously told the writer Michael Lewis: “The Democrats don’t matter. The real opposition is the media. And the way to deal with them is to flood the zone with shit.”15
The history of another term associated with Donald Trump—“fake news”—is instructive here. Originally, it described a very specific phenomenon: websites publishing false articles in the hope of getting clicks from social media and thus advertising dollars. The iconic example was the claim that the pope endorsed Trump’s presidential bid. When Trump won, for a while there was a moral panic, serious commentators worried that gullible voters had been lured into voting for Trump because they believed these outrageous lies.
That panic was a mistake. Academic studies found that fake news was never widespread or influential; most of it was consumed by a small number of highly conservative, elderly voters who were likely Trump supporters all along. These false stories quickly became much less of a problem, as social media websites woke up to the threat.16
But the idea of “fake news” became a powerful one—an excuse to dismiss any inconvenient claim from any source, a modern version of the cynical aphorism about “lies, damned lies, and statistics.” Mr. Trump, with his twisted talent for turning a complex issue into a political cudgel, deployed the term to demonize regular journalists. So did many other politicians, including Theresa May, then prime minister of the UK, and her opposite number, the Labour Party leader Jeremy Corbyn.
“Fake news” resonated because it tapped into an unfortunate truth: there is plenty of slapdash journalism even in mainstream outlets, as we shall see. But there are also serious and responsible journalists who carefully source their claims, and they found themselves being tossed into the same mental trash can as the pope-endorses-Trump merchants.
I worry about a world in which many people will believe anything, but I worry far more about one in which people believe nothing beyond their own preconceptions.
In the spring of 1965, a US Senate committee was pondering the life-and-death matter of whether to put a health warning on packets of cigarettes. An expert witness wasn’t so sure about the scientific evidence, and so he turned to the topic of storks and babies. There was a positive correlation between the number of babies born and the number of storks in the vicinity, he explained.17 That old story about babies being delivered by storks wasn’t true, the expert went on; of course it wasn’t. Correlation is not causation. Storks do not deliver children. But larger places have more room both for children and for storks. Similarly, just because smoking was correlated with lung cancer did not mean—not for a moment—that smoking caused cancer.
“Do you honestly think there is as casual a relationship between statistics linking smoking with disease as there is about storks?” asked the committee chair. The expert witness replied that the two “seem to me the same.”18
The witness’s name was Darrell Huff.
He’d been paid by the tobacco lobby to do what he did best: weave together witty examples, some statistical savvy, and a certain amount of cynicism to cast doubt on the idea that cigarettes are dangerous. He was even working on a sequel to his masterpiece—although it was never published. The sequel’s name was How to Lie with Smoking Statistics.19
Doubt is a powerful weapon, and statistics are a vulnerable target. That target needs defenders. Yes, it’s easy to lie with statistics—but it’s even easier to lie without them.*
And more important, without statistics it’s impossible to tell the truth—to understand the world so that we can try to change it for the better, like Richard Doll and Austin Bradford Hill. What they did took some insight and determination, but it required neither genius nor incomprehensible mathematical techniques. They counted what mattered—smokers, nonsmokers, cases of lung cancer, cases of heart disease. They counted them methodically and patiently, and, on the basis of the evidence they gathered, drew their conclusions with care. Over the years, those conclusions have saved the lives of tens of millions of people, perhaps including their own: after Hill gave up his pipe and joined Doll as a nonsmoker, both men lived into their nineties.
When we use statistics with assurance and wisdom, we see trends that would otherwise be too subtle to discern. The modern world is very big, very complex, and very, very interesting. Almost 8 billion people live here. Trillions of dollars change hands daily in our economy. In the typical human brain, there are 86 billion neurons.20 On the internet, there are around 2 billion websites. And a new virus can spread from a single person to thousands, millions—even billions—of others. Whatever we’re trying to understand about the world, one another, and ourselves, we won’t get far without statistics—any more than we can hope to examine bones without an X-ray, bacteria without a microscope, or the heavens without a telescope.
There’s a popular story about Galileo’s telescope: that even as the father of astronomy stood accused of heresy by the Roman Catholic Church, senior cardinals would not look through the instrument he had made, proclaiming it to be a magician’s trick. Galileo said he had seen mountains on the moon? Surely the lens of the telescope was dirty. He had seen the moons of Jupiter? Pah! The moons were in the telescope itself. They refused to look.
Four centuries later, it is easy for us to laugh at the story—which, by the way, has been exaggerated over the years.21 We shouldn’t be so self-satisfied. Many of us refuse to look at statistical evidence because we’re afraid of being tricked. We think we’re being worldly-wise by adopting the Huff approach of cynically dismissing all statistics. But we’re not. We’re admitting defeat to the populists and propagandists who want us to shrug, give up on logic and evidence, and retreat into believing whatever makes us feel good.
I want us to do something different. I want to give you the confidence to pick up the telescope of statistics and use it to scrutinize the world. I want to help you understand the logic behind statistical truths and escape the grip of the flawed logic, emotions, and cognitive biases that shape the falsehoods.
So look through the statistical telescope and gaze around. You will be amazed at how clearly you will be able to see.