A court of law may determine the meaning of replication in science
In the rough-and-tumble world of science, disputes are usually settled in time, as a convergence of evidence accumulates in favor of one hypothesis over another. Until now.
On April 10, 2006, the economist John Lott filed a defamation lawsuit against the economist Steven Levitt and HarperCollins, the publisher of Levitt’s 2005 book Freakonomics. At issue is what Levitt meant when he wrote that scholars could not “replicate his results,” referring to Lott’s 1998 book More Guns, Less Crime (University of Chicago Press). Lott employed a sophisticated statistical analysis on data from state-level variation in “carry and conceal” laws, finding that states that passed laws that permit citizens to carry concealed weapons saw statistically significant declines in robbery, rape, and homicide compared with states that did not pass such laws.
As is typical with such politically charged research, considerable controversy followed publication of Lott’s book, with a flurry of conference presentations and journal papers published, some of which replicated his results and some of which did not. For example, in a series of papers published in the Yale and Stanford law reviews (available at http://papers.ssrn.com), Lott and his critics debated the evidence.
In Freakonomics, Levitt proffered his own theory for the 1990s crime decline—Roe v. Wade. According to Levitt, children born into impoverished and adverse environments are more likely to grow up to become criminals. After Roe v. Wade, millions of poor, single women had abortions instead of future potential criminals; twenty years later the pool of potential criminals had shrunk, along with the crime rate. Levitt employed a comparative statistical analysis to show that the five states that legalized abortion two years before Roe v. Wade witnessed a crime fall earlier than the other forty-five states. Further, those states with the highest abortion rates in the 1970s experienced the greatest fall in crime in the 1990s. Finally, Levitt showed that three additional factors reduced the pool of criminals: increased rates of imprisonment, increased number of police, and the bursting of the crack cocaine bubble.
One factor that Levitt dismissed is Lott’s, in a single passage in the middle of a thirty-page chapter, in which he concluded that “Lott’s admittedly intriguing hypothesis doesn’t seem to be true. When other scholars have tried to replicate his results, they found that right-to-carry laws simply don’t bring down crime.”
According to Lott’s legal complaint, “the term ‘replicate’ has an objective and factual meaning” and that other scholars “have analyzed the identical data that Lott analyzed and analyzed it the way Lott did in order to determine whether they can reach the same result.” When Levitt said that they could not, he is “alleging that Lott falsified his results.”
I asked Levitt what he meant by replicate. “I used the term in the same way that most scientists do—substantiate results.” Substantiate, not duplicate. Did he mean to imply that Lott falsified his results? “No I did not.” In fact, others have accused Lott of falsifying his data, so I asked him why he is suing Levitt. “Having some virtually unheard of people making allegations on the Internet is one thing. Having claims made in a book published by an economics professor and printed by a reputable book publisher, already with sales exceeding a million copies, is something entirely different. In addition, Levitt is well-known and his claims unfortunately carry some weight. I have had numerous people ask me after reading Freakonomics whether it is really true that others have been unable to replicate my research.”
“Replicate” is a verb that depends on the sentence’s subject. “Replicate methodology” might capture Lott’s meaning, but “replicate results” means testing the conclusion of the methodology, in this case that more guns causes less crime. The problem is that many scientific experiments and statistical data sets are so complicated that the failure to replicate more likely indicates unconscious mistakes made during the original research or in the replication process, rather than conscious fakery.
Mr. Lott, tear down this legal wall and let us return to doing science without lawyers. Replicating results means testing hypotheses, not merely duplicating methodologies, and this central tenet of science can only flourish in an atmosphere of open peer review.
Update: A federal judge found in Levitt’s favor in the charge that his claim that scholars had not replicated Lott’s findings in Freakonomics was not defamatory; however, the judge found in Lott’s favor in a portion of the complaint involving a private email that Levitt sent to another economist about Lott’s research, leading to Levitt issuing a retraction letter. See http://bit.ly/1lC5AH5 and http://bit.ly/Y2j70a