A DEADLY GAME OF TELEPHONE: FROM THE RESEARCH LAB TO YOUR FACEBOOK FEED
Founder of TriangleBeWell.com and contributing author to Proteinaholic: How Our Obsession with Meat Is Killing Us and What We Can Do About It by Garth Davis, MD
One morning not so long ago, I woke up to find an unusual headline was going viral on my Facebook feed: “New Study Claims Eating Bacon May Prolong Your Life.” It took me about ten minutes of clicking from blogs to newspapers to press releases to find the actual study, which examined niacin supplementation in roundworms and had nothing to say about bacon or humans. What was fascinating—and horrifying—about this particular example of social media “clickbait” was the way it had clearly manipulated a set of facts to garner attention. Science took a backseat to ad revenue.
I believe most people want to make good decisions about their health. Unfortunately, it can be hard to make good decisions when we are constantly being bombarded by misinformation in the media. Some people end up chasing fads, ultimately adopting the fatalistic attitude that if we can’t know the truth, we might as well give up and enjoy ourselves.
Too often, this misdirection has to do with meat. In this essay, I’ll show you two recent examples of how poorly conducted studies fooled lazy or ill-informed journalists into making animal products seem a whole lot healthier than they actually are. Let the reader and eater beware!
The first study was published in 2014 with the headline “Association of Dietary, Circulating, and Supplemental Fatty Acids with Coronary Risk.” Soon after, Time magazine ran a cover story titled “Eat Butter: Scientists labeled fat the enemy. Why they were wrong.” The full-page image of an artfully curling pat of butter reinforced a tantalizing message: Advances in nutritional science had finally debunked the misguided (and depressing) notion that eating fat caused obesity and disease. And yet, the article produced no original research to back up this claim. In reality, it was a meta-analysis of seventy-two published studies that examined the associations between intake of different types of fat (polyunsaturated and saturated, principally) in different populations globally. The study’s conclusion was that the “current evidence does not clearly support cardiovascular guidelines that encourage high consumption of polyunsaturated fatty acids and low consumption of total saturated fats.”
From the beginning, it was clear Time magazine had mischaracterized the findings. The studies analyzed couldn’t prove that vegetable oil is better than butter and lard. Put another way, imagine a study that compared cigarette smoking to tobacco chewing, and found that smoking wasn’t much worse than chewing tobacco. A high school sophomore who wrote an essay called “Smoke Marlboros: Scientists labeled cigarettes the enemy. Why they were wrong,” would receive a failing grade. At the same time, the study wasn’t making a strong comparison. The only way to exonerate saturated fat was to compare it to unsaturated fat. A fairer and more meaningful analysis would have compared saturated fat to whole grains, or leafy greens, or potatoes. Anything else is a straw man argument.
Have you seen the acronym GIGO? It stands for “garbage in, garbage out”—exactly what happened here. The majority of studies the authors chose to include employed statistical sleight of hand to erase the strong associations between saturated fat consumption and coronary disease. This sleight of hand was accomplished using a technique called statistical adjustment. This can be a fine and appropriate thing to do when you’re comparing two groups that are different in meaningful ways. If you’re comparing death rates from cancer in the USA and Botswana, for example, you have to account for the fact that the Botswana population is much younger, on average, than the US population. The goal of statistical adjustment is to remove numerical noise, but it can hide the truth if used to adjust for factors that directly influence the outcome.
And that’s what the researchers in these studies did. They adjusted for calories consumed, blood pressure, body mass index (BMI), cholesterol, and fiber as well as intake of protein, carbohydrates, and vegetables. In other words, they were able to erase all differences between groups eating a lot and a little saturated fat by erasing all the ways saturated fat contributes to heart disease. Saturated fat contributes to heart disease by increasing calories consumed (meat and dairy are very calorically dense foods compared to fruits and vegetables), raising blood pressure by clogging arteries, increasing BMI, raising cholesterol, decreasing fiber intake by crowding out plant foods that contain fiber, increasing protein consumption (animal fat and protein go together in the diet), and decreasing consumption of carbohydrates and vegetables. Adjusting for these causal mechanisms is like adjusting for the velocity and weight of the bullet to prove that AK-47s aren’t any deadlier than airsoft rifles.
The studies included in the meta-analysis were chosen specifically because they looked at groups eating more or less the same diet. And to make sure saturated fat got a clean bill of health, the author of this study evened out any remaining differences by using quintile transformations. For example, one study compared Japanese people who ate varying amounts of saturated fat, and another study looked at the same variation among people in Finland. In both cases, there was no correlation. Here’s why: The Japanese people all ate small amounts of saturated fat. Even the biggest consumers in Japan ate much less than the lowest consumers in Finland. So when the two data sets were graphed and adjusted, they looked identical. But, if you compare Japanese to Finns, you get a totally different story. The Japanese rate of death from coronary disease was about 51 people per 100,000 people per year, while that of the Finns was 244, almost five times greater.
Similarly, a 2014 study titled “Effects of Low-Carbohydrate and Low-Fat Diets: A Randomized Trial” garnered serious headlines in the New York Times. The paper of record’s lede on the topic was: “People who avoid carbohydrates and eat more fat, even saturated fat, lose more body fat and have fewer cardiovascular risks than people who follow the low-fat diet that health authorities have favored for decades, a major new study shows.” Once the Times gave the study the stamp of legitimacy, other news outlets rushed to spread the good news that bacon and butter are better for us than chard and kale.
This study aims to be the gold standard of biomedical research, a randomized clinical trial. It took 148 people and randomly put them in a low-carb or a low-fat group and monitored their weight and a few cardiovascular risk factors (mostly their lipid panels). After a year, the low-carb group lost more weight and had better lipids than the low-fat group.
But should we all embrace high-protein, animal-based diets if we want to live healthier lives? Or is something else going on here? Let’s peek under the hood of this study.
The biggest problem with this study is the definition of low-carb and low-fat. When the researchers define low-fat as “less than 30 percent of calories from fat,” I have to invoke Mandy Patinkin’s immortal line from The Princess Bride: “You keep using that word. I do not think it means what you think it means.”
A truly low-fat diet would limit fat to 10 to 20 percent of calories, at the very most. Given that the standard American diet is about 35 percent fat, you can see that very little fat has to be eliminated to adhere to this “low-fat” label.
The low-carb group was instructed to consume less than 40 grams of carbohydrates daily. Given that no one has ever seen a carbohydrate, it’s not surprising that they didn’t achieve this goal. Actually, they exceeded it by over 200 percent, consuming an average 127 grams of carbs daily.
So we’re drawing conclusions based on a comparison of a low-fat group that didn’t eat low fat with a low-carb group that didn’t eat low carb. And the New York Times wants us to pay attention to this?
We’re left with a question: How come the supposed low-fat group didn’t lose as much weight as the alleged low-carb group? Although the discussion within the published article claimed that “total energy intake was similar between groups,” the reported data contradict this assessment. The low-fat group actually took in an average of 79 more calories per day than the low-carb group: 1,527 vs. 1,448.
Over the course of a year, that means the low-fat group consumed almost 29,000 more calories than the low-carb group. Given that it takes 3,500 calories to add a pound of fat, we can see that the low-fat group could be expected to gain 8.24 more pounds than the low-carb group simply based on consumption. Because the study reported that the difference in weight loss was only 7.7 pounds, this is explainable entirely based on the difference in calories. If The New York Times wishes to issue a retraction, may I suggest a better headline for this study: “Eating More Calories Increases Weight and Cardiovascular Risk Factors.”
This essay is far too short to give you all the tools necessary to recognize scientific mumbo jumbo when you see it. In general, though, if something appears too good to be true or contradicts prevailing wisdom, it’s probably not worthy of breathless media coverage. Only after that study has been replicated a dozen times, especially by scientists skeptical of its conclusions, should it appear in our newspapers and our Facebook feeds.