Tarleton Gillespie
In the wake of the 2016 U.S. presidential election, facing criticism over the prevalence of fake news, Facebook first proposed a trio of measures to address the problem: (a) clarifying part of the existing flagging system, marking “illegal, misleading, and deceptive content,” to encourage users to flag news articles they felt were fraudulent; (b) enlisting fact-checking organizations like Snopes, FactCheck, and PolitiFact to assess some of these flagged news articles and mark them “disputed”; and (c) promising to pursue artificial intelligence (AI) techniques that might automatically discern the fraudulent from the genuine.1
Let’s set aside not only the limitations of these responses, but also the profound problems of defining “fake news,” of distinguishing hoaxes and falsehoods from propaganda and spin.2 What I find revealing is that Facebook looked at the problem of fake news and saw it as a problem of content moderation, choosing as its first response to deploy its already-existing content moderation apparatus, or a souped-up version of it.
Every major social media company has built up a specific apparatus for content moderation.3 Most take a customer service approach: users (and, increasingly, software) are tasked with identifying problematic content or behavior; platform moderators then engage in a procedural review, behind the scenes; they then decide to remove that content or not, based on their own guidelines and judgment. These procedures, built over more than a decade, have arguably been good enough for the needs of the platforms, in the sense that they have been able to scale to millions (or even billions) of users, to maintain enough promise of a healthy community that enough users regularly return, and to have kept up with new and more troubling phenomena as they emerged.
That, shall we say, is the best version of the story. Another is that a toxic culture of harassment, especially targeting women and minorities, appears to have rooted itself in social media, blithely tolerated by platform managers keen on encouraging their own ideas of free speech and profiting from the data they collect along the way; legislators in Europe and elsewhere are demanding more stringent interventions from social media platforms when it comes to hate speech and terrorist propaganda; and a flood of fraudulent news and conspiracy theories have eaten away at the public trust and influenced the way voters think, sufficiently perhaps to have had an effect on national elections. Even a high bar for success when moderating at this scale still allows hundreds of thousands of errors, and hundreds of thousands of oversights—each of which represents a user being wronged or left unprotected.
This system is starting to crack at the seams, and public disaffection for content moderation is joining other public concerns, about data privacy, targeted advertising, and the impact on journalism, in a fundamental reconsideration of the responsibility of platforms. Nevertheless, it is the system that’s in place, firmly settled as the way platforms distinguish what they want to keep from what they want to remove.
Once a platform has a mechanism for moderation in place, it will tend to use it, regardless of whether or not it is suited to the new problem. This is deeper than just “when you have a hammer, every problem looks like a nail.” Facebook has deeply invested in a complex, sociotechnical apparatus for moderating, with a complex division of a very large labor force, as its answer to an array of problems—one very particular kind of answer. It is compelled to deploy it whenever possible, for whatever problem; in fact, it comes to see the problem through its lens. But whatever your assessment of how effectively platforms moderate, fake news presents a new kind of challenge to platforms. There are some kinds of problems that simply cannot be cleaned up, because they are systemic, because the platform and the entire information ecosystem play into their circulation, because they reflect the very nature of the platform itself.
Early efforts at content moderation were designed with two kinds of problems in mind. The first were deliberate and obvious violations of the platform’s guidelines, by those who rejected the rules or enjoyed mucking with the system: porn posted to platforms that clearly forbid it, trolls who invade communities to wreak havoc, Wikipedia vandals. These violations are obvious, in that the breach of the rules is clear, even if finding each and every violation isn’t so simple. As the trolls got more sophisticated, the challenge of detection grew: 4chan-ers posting seemingly innocent videos with porn scenes embedded inside, or harassers engaged in organized efforts to swarm a victim without warning.
Platforms built their early moderation tools to keep the porn and trolls out. Obvious violations are politically easy, in part because they match and reassert the implied (or assumed) consensual norms of the community. But platforms quickly found that they faced a second kind of moderation challenge: instances in which the content or behavior in question was contested, in which there was no consensus about community norms. One user finds a photo offensive, another does not. One user feels harassed, the other believes it is all in good fun. One sees their speech as legitimately political, while their critics find it hateful. Sometimes these instances figured around a rule that was itself contested, or an exception that only some found reasonable: nudity policies that didn’t suit women who felt that, if they could breastfeed in public, they should be able to do so on Facebook too; drag queens who felt they had the right to profiles using their stage names. Platforms reluctantly found themselves adjudicating their own policies around hotly contested cultural issues, weighing the opinions of users with different (even incompatible) beliefs, ethics, and cultures.
Platform moderation was designed to handle both kinds of moderation: to police the obvious violations and mediate the contested ones. But platforms increasingly face a third category: parasitic content.4 Here I include misinformation, disinformation, propaganda—the kind of contributions that the unwieldy term fake news attempts to capture.5 By “parasitic” I mean those contributions that are aware of the workings of the platform, and are designed with that awareness in mind—constructed to appear “genuine”6—but they take advantage of the circulation, legitimacy, and context the medium offers to do something beyond (or even counter to) its apparently genuine aims.
Like traditional forms of propaganda, fake news is designed to take advantage of the way the particular system works—procedurally, algorithmically, and financially. Fake news exploits platforms by simulating the very things platforms want to circulate, the very things platforms are optimized for.7 If a fraudulent piece of propaganda can look just like news, then maybe some users will forward it like news. If it looks like news that aligns with the user’s political beliefs, even better: maybe it will be shared as evidence of that political belief, as a performance of that political identity, as a badge of membership in that political tribe.8 And if it looks like the kind of provocative, alluring, eminently clickable news that these platforms now thrive on, the kind that drives the “engagement” that is the economic imperative of commercial platforms, then platforms may be reluctant to remove it. Parasitic content that manages to circulate, or go viral, enjoys whatever patina of legitimacy that the platform offers other content that goes viral: appearing popular, hot, zeitgeist, or newsworthy.
Addressing parasitic content is fundamentally unlike the kinds of content moderation that these platforms traditionally engage in. Fake news represents an “existential crisis” for social media platforms,9 because it understands the platform and its incentives, and turns the platform against itself in ways that it will be reluctant to correct. Sometimes the developers of fake news understand the mechanics and incentives of the system even better than the platform operators themselves. Like a virus, fake news thrives by understanding its host, using the system against itself. And, like a virus, rendering it inert may require altering the nature of the host itself—even the risk of killing it.
If that’s not worrisome enough, let’s acknowledge an ever more difficult truth that fake news makes plain. It’s not just that someone with ill intentions can design a page to look like a news site or write a catchy headline. Parasitic content can emulate “genuine” content so well because all content on social media platforms is, to some degree, tactical. In the broadest sense, all contributions to a platform attempt to take advantage of that platform, in that all contributions have aim and purpose. We flatter ourselves when we presume a simple distinction between (our) genuine contributions and (their) devious ones: all users want their content to circulate—we are invited to communicate in this way—and all users seek to understand and exploit the system so that might happen, if only in little ways. Parasitic does not necessarily mean “nefarious,” but it does reveal the opportunity that fake news exploits.
Much depends on who has access to different kinds of tools on the platform, and who enjoys a more or less sophisticated understanding of the system: I may know that including a hashtag in my post may help it circulate more widely or be found by more users; a newspaper partnered with a platform may be given more powerful tools and privileges that allow it to manage distribution in more sophisticated ways than I can; an advertiser or political campaign given access to personalized data has a still greater ability to utilize that system for its own ends. But in every case, the content has been outfitted to better exploit the system: make it as clickable as possible; work the system so it is more likely to be liked, retweeted, or forwarded; understand and optimize for the algorithms that pick and choose. Fake news not only emulates the look and feel of legitimate news; it emulates the tactics of legitimate news outlets, of all users, when we compete for attention in social media.
Facebook and Google did take other steps, besides content moderation, to combat fake news in ways that were more sober about the existential threat it posed, that acknowledged the way it took advantage of the platform’s incentive structure. In November 2016, Facebook and Google removed what they deemed to be fraudulent providers from their advertising networks.10 Their aim was to prevent them from making a quick profit from the advertising, thereby removing one of its most powerful incentives. Posts written by Macedonian teenagers, meant not to drive a political wedge into the U.S. electorate but to make a quick buck from an initial burst of “can you believe it?!” clicks and repostings, saw their ad revenue disappear.11
This was, of course, not a complete solution either. It only addressed those that were economically motivated; those motivated by politics would not likely be discouraged by this change. Still, it was a powerful step toward addressing the existential crisis of parasitic content. It recognized that the structural design of these ad networks offered a powerful incentive to the production of fake news, and more fake news was being produced to take advantage of that incentive.12 Fake news has led platform managers to start acknowledging a harder truth: not just that there are harms on their platforms, or even that they might be held responsible for them in a legal sense, but that their platforms may in some way facilitate, encourage, entice, amplify, and profit from these harms. The platforms are not just available for misuse; they are structurally implicated in it.
Many commentators struggle to find the language for how platforms are implicated in their own misuse. We remain enamored with the idea that technologies are mere tools and that any detrimental consequences that follow do not fall at their feet. The simplest case to make is that social media platforms provide the tools to speak at zero cost, draw people into closer proximity, flatten the obstacles between them, deliver those contributions to others, allow them to persist, and organize them to be found. More perniciously, social media platforms curate users’ contributions algorithmically, preferring some over others, according to specific logics of popularity and personalization. They reward popularity and virality and exaggerate its effects, and they reward homophily while offering the illusion of a gathered public.13
All this makes platforms ripe for fake news, conspiracies, and hoaxes. It’s more than the mere fact that misinformation can circulate alongside true claims or that unvetted users can look like professional experts. Other tactics are available. Those looking to mislead can produce a whole lot of misleading content on the cheap, post it, and see what sticks. Popularity produces pathways of circulation that move their claims, and give them added social legitimacy.14 Where disinformation used to require either highly constructed media campaigns or lateral networks of rumor, now individuals can try throwing a little grit into the system, to see what disrupts the most. Platforms amplify small interjections, when they seem like something worth sharing. And those deeply invested in such misinformation can produce not only the misleading claims, but whole networks of people and bots to circulate it and give the appearance of authentic popularity—an additional signal that platforms identify and use as justification to amplify further.
Fake news is fundamentally unlike porn, harassment, and hate speech, in that it is designed to emulate exactly what the platform wants to distribute most. But there are other phenomena that we might call parasitic content, and platforms have already grappled with them in ways more suited to the challenges they pose: spam, search engine optimization (SEO) tactics, fake accounts, distributed denial of service (DDoS) attacks, clickbait, hoaxes, and bots. While these challenges have also, at times, been treated like content moderation problems, these “shady practices” pose much the same existential crisis that fake news does.15
Google and other search engines are in a constant battle against web designers trying to make their sites recognizable to the indexes, in the hopes that the site will be amplified as more relevant than it is otherwise. An entire industry, SEO, developed to pursue this end, promising to understand what Google’s algorithms “want” in order to ensure that its clients’ pages would be the lucky recipient of that desire.16 In this battle, Google asserts the right to demote sites that don’t play by its rules—a right that is sometimes disputed by those that are demoted, but rarely by search users. Like fake news, SEO attempts to understand the technical and incentive structure of search, and simulate features that will most likely lead search engines to deem it relevant. And like fake news, effective SEO threatens to undermine the very premise of search engines—if the top results on Google are there because those sites best approximated what the search engine looks for, of what value are those results?
Similarly, when Facebook began to struggle with clickbait, the question was whether Facebook’s tendency to reward tantalizing headlines was having some aggregate damage on the quality of news, was incentivizing the wrong kind of information production. Like fake news, clickbait attempts to take advantage of the workings of the platform as a sociotechnical system. Because the mechanism for circulation is whether users click, packaging content in ways that maximized the appeal at that moment and in that form was strategically valuable for information providers who wanted readership, thrived on vitality, or simply had to make money in a moment when social media was becoming an increasingly dominant environment for news.17
Unlike Google’s efforts to keep ahead of SEO, Facebook’s efforts to address clickbait were at times controversial. News publishers complained that clickbait sites were getting an unfair advantage; when Facebook adjusted its newsfeed algorithm to privilege “authentic content,” those sites complained that Facebook was picking winners and losers with an algorithm that should represent a level playing field.18 The fortunes of specific outlets rose and fell as Facebook calibrated its algorithm to disincentivize the worst forms of clickbait, which it had itself called into existence.19 (And, as with fake news, it is not as if “legitimate” journalism doesn’t also engage in an effort to lure readers with an eye-catching headline or an exaggerated promise of what they might find in the article.20 All news is parasitic in this way. These are all questions of degree and legitimacy.) Clickbait is, in many ways, a forbear of the fake news problem; it’s not surprising that Facebook’s response to fake news was continuous to its efforts to limit clickbait.
But more than anything, fake news may have more in common with spam. As spam is to e-mail, fake news is to social media: both are, as Finn Brunton put it, “the use of information technology infrastructure to exploit existing aggregations of human attention.”21 Spam wants to be circulated as a legitimate commercial appeal; fake news wants to be circulated as legitimate political fact. Spam often emulates (to varying degrees of sophistication) other familiar commercial, human, or institutional appeals: a bargain price for a desired product, a plea for assistance from a person in need, a customer service alert from a trusted bank. Some is amateur in its simulation and relatively easy to detect, while some is sophisticated enough to trick even a savvy user into clicking. And the economics are much the same: like fake news, spam doesn’t have to trick everyone; it’s a game of numbers, where even if a small percentage is fooled, the impact may be sufficient.
Traditional content moderation approaches, like tasking users with identification, making internal determinations of what should and should not circulate, and developing software to automate the detection of misinformation, are ill fitted to this existential threat. Fake news might not be something that can be policed away like porn. It not only simulates the news, and produces activity that simulates popularity; it makes plain that the system is already designed for tactical persuasion, already designed to reward the clickable over the true. Shutting off the flow of advertising revenue to fraudulent information providers was a powerful response; rather than policing away fake news, this changed the very terms on offer from the platform, undercutting the structural incentive that called it forth.
Perhaps it is possible to address fake news, not as bad content to be moderated away, but as a fundamental violation of the premise itself. Like spam, parasitic content must be treated as a conceptual violation of the premise of platforms themselves, rooted out for its own sake. It “provokes and demands the invention of governance and acts of self-definition on the part of those with whom it interferes.”22 Like SEO, it must be justified that this intervention is not editorial, but foundational. And as with clickbait, it must be disincentivized by changing the economic, political, and material structures and rewards that encourage it.
But, and this is the hardest part, it may require a sober rethinking of what Facebook really is, what Google really is; if these are in fact systems that value engagement over all other values, aspects of these underlying premises may need to change for the current flood of fake news to dwindle. “That is, the channel should be understood in terms of its capacity to fail, in the sense of being subject to a variety of parasites.”23 Social media platforms will have to decide whether fake news represents a violation of a core principle—perhaps one that was not well formed or articulated, like civic virtue or public collective obligation—or the unavoidable outcome of a principle too foundational to reconsider.
1. Casey Newton, “Facebook Partners with Fact-Checking Organizations to Begin Flagging Fake News” The Verge, December 15, 2016, http://www.theverge.com/2016/12/15/13960062/facebook-fact-check-partnerships-fake-news.
2. On the responses’ limitations, see Robyn Caplan, Lauren Hanson, and Joan Donovan, Dead Reckoning: Navigating Content Moderation after “Fake News” (New York: Data and Society Research Institute, 2018). On defining “fake news,” see Nicholas Jankowski, “Researching Fake News: A Selective Examination of Empirical Studies,” Javnost—The Public 25, nos. 1–2 (2018): 248–255, https://doi.org/10.1080/13183222.2018.1418964; Johan Farkas and Jannick Schou, “Fake News as a Floating Signifier: Hegemony, Antagonism and the Politics of Falsehood,” Javnost—The Public 25, no. 3 (2018): 298–314; On distinguishing hoaxes from propaganda, see Gilad Lotan, “Fake News Is Not the Problem,” Data and Society, November 16, 2016, https://points.datasociety.net/fake-news-is-not-the-problem-f00ec8cdfcb.
3. Tarleton Gillespie, Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media (New Haven, CT: Yale University Press, 2018).
4. Playing lightly off Michel Serres’s idea of the “parasite.” See Michel Serres, The Parasite, trans. Lawrence Schehr (Baltimore: Johns Hopkins University Press, 1982); also Paul Kockelman, “Enemies, Parasites, and Noise: How to Take Up Residence in a System without Becoming a Term in It,” Journal of Linguistic Anthropology 20, no. 2 (2010): 406–421. Thanks to Dylan Mulvin for pointing me in this direction.
5. Caroline Jack, “Lexicon of Lies: Terms for Problematic Information,” Data and Society, August 9, 2017, https://datasociety.net/output/lexicon-of-lies/.
6. Alexis Madrigal, “Why Facebook Wants to Give You the Benefit of the Doubt,” The Atlantic, July 19, 2018, https://www.theatlantic.com/technology/archive/2018/07/why-facebook-wants-to-give-you-the-benefit-of-the-doubt/565598/.
7. Tarleton Gillespie, “Algorithmically Recognizable: Santorum’s Google Problem, and Google’s Santorum Problem,” Information, Communication and Society 20, no. 1 (2017): 63–80.
8. Francesca Polletta and Jessica Callahan, “Deep Stories, Nostalgia Narratives, and Fake News: Storytelling in the Trump Era,” American Journal of Cultural Sociology 5, no. 3 (2017): 392–408.
9. Natasha Lomas, “Fake News Is an Existential Crisis for Social Media,” TechCrunch, February 18, 2018, http://social.techcrunch.com/2018/02/18/fake-news-is-an-existential-crisis-for-social-media/.
10. Kaveh Waddell, “Facebook and Google Won’t Let Fake News Sites Use Their Ad Networks,” The Atlantic, November 15, 2016, http://www.theatlantic.com/technology/archive/2016/11/facebook-and-google-wont-let-fake-news-sites-use-their-ads-platforms/507737/.
11. Craig Silverman and Lawrence Alexander, “How Teens in the Balkans Are Duping Trump Supporters with Fake News,” BuzzFeed News, November 3, 2016, https://www.buzzfeed.com/craigsilverman/how-macedonia-became-a-global-hub-for-pro-trump-misinfo.
12. Allegedly, Facebook had considered an upgrade to its newsfeed algorithm before the election that would have identified fake news / hoaxes, but because it appeared to disproportionately single out right-wing sites, it was shelved for fear of appearing politically biased; Michael Nunez, “Facebook’s Fight against Fake News Was Undercut by Fear of Conservative Backlash,” Gizmodo, November 14, 2016, http://gizmodo.com/facebooks-fight-against-fake-news-was-undercut-by-fear-1788808204; Josh Constine, “Facebook Chose to Fight Fake News with AI, Not Just User Reports,” TechCrunch, November 14, 2016, https://techcrunch.com/2016/11/14/facebook-fake-news/.
13. Ben Tarnoff and Moira Weigel, “Why Silicon Valley Can’t Fix Itself,” The Guardian, May 3, 2018, https://www.theguardian.com/news/2018/may/03/why-silicon-valley-cant-fix-itself-tech-humanism.
14. Karine Nahon, and Jeff Hemsley, Going Viral (Cambridge, UK: Polity Press, 2013).
15. Malte Ziewitz, “Shady Cultures,” Theorizing the Contemporary, Cultural Anthropology, April 28, 2017, https://culanth.org/fieldsights/shady-cultures.
16. Dipayan Ghosh and Ben Scott. “Digital Deceit: The Technologies behind Precision Propaganda on the Internet,” New America, January 2018, https://www.newamerica.org/public-interest-technology/policy-papers/digitaldeceit/.
17. Zizi Papacharissi, “The Importance of Being a Headline,” in Trump and the Media, ed. Pablo Boczkowski and Zizi Papacharissi (Cambridge, MA: MIT Press, 2018), 71–77.
18. Will Oremus, “Facebook Is Cracking Down on Inauthentic Content,” Slate, January 31, 2017, http://www.slate.com/blogs/future_tense/2017/01/31/facebook_is_cracking_down_on_inauthentic_content_in_the_news_feed.html.
19. Robyn Caplan and danah boyd, “Isomorphism through Algorithms: Institutional Dependencies in the Case of Facebook,” Big Data and Society 5, no. 1. (2018), https://doi.org/10.1177/2053951718757253.
20. Kalev Laeetaru, “The Inverted Pyramid and How Fake News Weaponized Modern Journalistic Practice,” Forbes, December 10, 2016, http://www.forbes.com/sites/kalevleetaru/2016/12/10/the-inverted-pyramid-and-how-fake-news-weaponized-modern-journalistic-practice/.
21. Finn Brunton, Spam: A Shadow History of the Internet (Cambridge MA: MIT Press, 2013), 199.
22. Ibid., 203.
23. Kockelman, “Enemies, Parasites, and Noise,” 412.