There’s a war out there, old friend. A world war. And it’s not about who’s got the most bullets. It’s about who controls the information. What we see and hear, how we work, what we think . . . it’s all about the information!
—COSMO, in Sneakers
THEY SAY NECESSITY IS THE MOTHER of invention. For Chad Hurley, Steve Chen, and Jawed Karim, that necessity was seeing Janet Jackson’s nipple.
During the live television broadcast of the 2004 Super Bowl, superstars Janet Jackson and Justin Timberlake teamed up for a duet at halftime, coming onstage soon after a salute to the troops fighting in Iraq. At the very moment that Timberlake sang “Bet I’ll have you naked by the end of this song,” he reached across to tear off a piece of Jackson’s top. For nine-sixteenths of a second, her bare breast stood exposed to the Houston air—and to 140 million viewers. What followed was dubbed “Nipplegate,” weeks of cultural commentary and hand-wringing about small children and stolen innocence. The Federal Communications Commission was flooded with a record-shattering 540,000 complaints. America Online, which had spent $10 million to sponsor the show, demanded its money back.
Yet as much as everyone was talking about what had happened, it was nearly impossible to find the uncensored evidence. Newspapers weren’t showing it. Networks weren’t showing it. And angry, soon-to-be-irrelevant America Online certainly wasn’t showing it. The videos of Jackson’s “wardrobe malfunction” were out there; they just needed to be hosted in one searchable, shareable place.
And so YouTube was born.
Yes, the website that would become the video archive of the human race was launched by an errant nip-slip. Yet the strangest part of the story wasn’t how unusual it was, but rather how typical. Nearly all of today’s internet giants can be traced to such beginnings. Facebook was a product of the juvenile, hacked-together Facemash. Twitter was the result of a hasty pivot from a failing podcasting service, originally marketed to the San Francisco rave scene. Even Google started with two Stanford nerds just trying to write a half-decent dissertation.
This is the DNA of the social media ecosystem: nearly universally male, white, drawn from America’s upper middle class, and dedicated, at least initially, to attacking narrow problems with equally narrow solutions. Despite their modest, usually geeky origins, these founders now rule digital empires that dictate what happens in politics, war, and society at large. It has been an uneasy reign as they come to grips with what it means to rule their kingdoms.
There’s no historical analogue to the speed and totality with which social media platforms have conquered the planet. The telegraph was pondered for at least two generations, while the internet gestated for decades in U.S. government labs. Beyond science fiction and the grand predictions of a few sociologists, social media was something that simply wasn’t—until suddenly it was.
The surprise is most obvious among the creators themselves. As their platforms graduated to their first thousand, million, and even billion users, these bright young founders weren’t pondering how their systems might be used to fight and win wars; they were mostly struggling to keep the lights on. More users required more servers. More servers required more investors. More investors required a sustainable business or, failing that, even more users. “This is the original sin of Silicon Valley,” writes design ethicist Mike Monteiro. “The goal of every venture-backed company is to increase usage by some metric end over end until the people who gave you that startup capital get their payday.”
Everything about these services was designed with growing the business in mind, engineering toward more users, and drawing them more deeply into the online experience. Consider something as innocuous as the “notification” icon—the red dot that has haunted the Facebook app for a decade. No part of the design is an accident. Red is the color of agitation and psychological arousal, the mere glimpse of which can lead to a spike in heart rate. It feels good to make red things go away. Because notifications are purposefully vague until touched, following them can feel like opening a present. (Is the notification a long, heartfelt comment from a close friend, or just another forgotten acquaintance’s birthday?) While the notification icon was certainly intended to make Facebook users’ lives easier, it was also intended to keep users buried in the app—part of what leads the average person to touch their phone 2,617 times each day. In short, that red button provides the good news of the company’s value to hand to Facebook shareholders at their annual meeting. Although Facebook engineers were essentially putting a drug in users’ pockets, it wasn’t their—or anyone’s—job to consider the potential side effects.
This single-minded push for growth was best illustrated by an internal memo circulated among Facebook leadership in the summer of 2016, at the very same time Russian propagandists were running rampant on the service and the Trump campaign was poring over tens of millions of ill-gotten Facebook profiles. “We connect people. Period,” a senior Facebook vice president wrote. “That’s why all the work we do in growth is justified. All the questionable contact importing practices. All the subtle language that helps people stay searchable by friends. All of the work we do to bring more communication in. The work we will likely have to do in China some day. All of it . . . Maybe it costs someone a life by exposing someone to bullies. Maybe someone dies in a terrorist attack coordinated on our tools.”
This Silicon Valley myopia would matter less if services like Facebook, YouTube, and Twitter were simply inventions, like the telegraph or radio, which might be repurposed by other tech pioneers while the original creators enjoyed fat royalty checks. But they’re not. These companies aren’t inventions but platforms—services that deliver the most value to users who visit them frequently, often addictively.
A good parallel to the present-day titans isn’t Samuel Morse, tinkering away in his workshop to build one of several competing telegraph services. Instead, it’s Alexander Graham Bell, whose Bell Telephone Company (later AT&T) would control virtually all telephone wire in the United States, under the motto “One Policy, One System, Universal Service.”
Scale allows the most successful Silicon Valley entrepreneurs to rule like absolute sovereigns over their platforms—and, by extension, over everyone who relies on them. If Mark Zuckerberg authorizes a small tweak to Facebook’s design—like replacing comment boxes with bubbles—the change will be seen by more than 2 billion users, immediately ranking it among the largest collective experiences in human history. In turn, tiny, imperceptible shifts in the newsfeed algorithm can turn previously niche media outlets into hulking behemoths while wrecking the fortunes of others. It can even, as we’ve seen, alter the course of American elections and of wars in the Middle East.
In some ways, we’re lucky that these mighty figures have chosen to rule their empires like benign and boring figureheads. They’re almost universally progressive, pledging themselves to the cause of social justice, but also striving to be militantly inoffensive in their public statements. They’ve established rules and regulations that seek to mirror or exceed the permissive speech codes of the United States. “In the process of building private communities,” writes John Herrman, “these companies [put] on the costumes of liberal democracies.” In so doing, however, they’ve avoided reckoning with the extent of their own burgeoning influence and power.
Part of the reason is an inherent contradiction. For all the talk of “community,” these platforms are businesses. Their “boss” isn’t the United Nations or even their user base; it’s shareholders. At the end of the day, the metrics that matter most aren’t the number of violent crimes averted or the number of humans shielded from harm; they are stock price and year-over-year revenue. In turn, for all the transparency these companies have forced upon the world, their most important decisions still originate in closed corporate boardrooms.
There’s also the technocratic, optimistic worldview that comes from companies staffed mostly by engineers working hard to build products. “You’re so focused on building good stuff,” explained Mike Hoefflinger, a former Facebook executive and author of Becoming Facebook, “you’re not sitting there thinking, ‘If we get lucky enough to build this thing and get two and a quarter billion people to use it, then this other bad stuff could happen.’”
Finally, the companies are haunted by a very real, deep cultural tension. Most of the people who build and maintain these politically pivotal platforms don’t particularly like politics. This ethos was on clear display at Hacker News, a popular Silicon Valley forum board, when “Political Detox Week” was announced shortly after the 2016 U.S. presidential election. As the rest of the country struggled to come to terms with the election results, an administrator declared:
Political stories are off-topic. Please flag them. Please also flag political threads on non-political stories. For our part, we’ll kill such stories and threads when we see them. Then we’ll watch together to see what happens.
Why? Political conflicts cause harm here. The values of Hacker News are intellectual curiosity and thoughtful conversation. Those things are lost when political emotions seize control. Our values are fragile—they’re like plants that get forgotten, then trampled and scorched in combat. HN is a garden, politics is war by other means, and war and gardening don’t mix.
In other words, this was a forum for “intellectual curiosity and thoughtful conversation,” and to this community of tech geeks, politics was by definition the opposite of that. Thus, at the very moment the forum members’ work had been shown to be remaking the political landscape, they felt they could just tune it out.
This “engineering first” mentality applies to both problems and potential solutions. Whenever these companies have had to reckon with a political, social, or ethical dilemma—ironically spawned by their platforms’ very success—they often grasp for another new technology to solve it. As a senior executive at one of these companies put it to us, “If we could use code to solve all the world’s problems, we would.”
But for all the reasons outlined in this book, the excuses have begun to fall short. It is one thing to run a social media service whose biggest headaches are copyright infringement and naughty pictures; it is quite another when that same service is being used to abet terrorism, stoke racism, and shatter entire political systems. When Mark Zuckerberg entertains pleas from Ukrainian activists to break a Russian “informational blockade” or dispatches Facebook engineers to “ensure the integrity of the German elections,” he is no longer simply the custodian of a neutral platform. When he vows that his company will devote itself to “spreading prosperity and freedom” or “promoting peace and understanding,” he is no longer simply a tech CEO. He is a new kind of leader, who has begun moving, reluctantly, to claim his position on the world stage.
Ultimately, the greatest challenge that confronts these social media giants has nothing to do with software code. It is a problem of corporate incentives, clashing cultures, and a historic revolution that has left both politics and Silicon Valley reeling. It is a problem of bestowing carefree engineers uninterested in politics with grave, nation-sized political responsibilities.
And although this is a problem with endless dimensions, at its heart it has always revolved around the same three questions: Should these companies restrict the information that passes through their servers? What should they restrict? And—most important for the future of both social media and the world—how should they do it?
Naturally, it’s another story that begins with the desire to see someone’s breasts.
“Marketing Pornography on the Information Superhighway: A Survey of 917,410 Images, Descriptions, Short Stories, and Animations Downloaded 8.5 Million Times by Consumers in over 2000 Cities in Forty Countries, Provinces, and Territories.”
When it was published in the Georgetown Law Journal in June 1995, Marty Rimm’s remarkably titled study became an overnight sensation. Rimm concluded that more than four-fifths of the nascent internet consisted of pornography, which the author claimed to have exhaustively cataloged. The finding was reported in every major newspaper, debated across TV and talk radio, and featured on the cover of Time magazine with a picture of a shocked toddler looking at a computer screen emblazoned with the new word “CYBERPORN.”
In many ways, it’s fitting that this instantly viral report was also a total fabrication. Marty Rimm was an attention-seeking Carnegie Mellon University student who’d gotten his work published by dodging peer review. Rimm did have another publication under his belt, The Pornographer’s Handbook: How to Exploit Women, Dupe Men, and Make Lots of Money, which suggested he wasn’t being entirely sincere in his statements about the pornographic menace. As his work came under scrutiny and his claims that the internet was almost completely porn were debunked, Rimm vanished and eventually changed his name.
Nonetheless, the damage had been done. While 1995 was a banner year for the internet, as the U.S. government officially relinquished control and millions of new users jumped online, in Congress this excitement gave way to moral panic. Thanks to one viral story and an audience of mostly tech-illiterate legislators, the internet now meant only one thing: pornography.
“The information superhighway should not become a red-light district!” bellowed Senator James Exon, an elderly Democrat from Nebraska. He introduced the Communications Decency Act (CDA), which made it a crime to “send to or display in a manner available to” those under 18 years of age any communication that depicted “sexual or excretory activities . . . regardless of whether the user of such service placed the call or initiated the communication.” The punishment was two years in prison and a fine of $100,000. In 1996, the law passed with overwhelming bipartisan support.
The final law had one crucial tweak, however. Two younger U.S. representatives—Chris Cox, a Republican from California, and Ron Wyden, a Democrat from Oregon—realized that unless something was done to protect websites that tried to police their own networks, the entire internet would be paralyzed by fear of lawsuits and prison time. Their consequent amendment became 47 U.S.C. § 230 (1996), known as Section 230. It was, in the words of Wired magazine, “the most important law in tech.”
Section 230 provided “protection for ‘Good Samaritan’ blocking and screening of offensive material,” essentially ruling that no website could be held accountable for the speech of its users. And no website that made a “good-faith” effort to enforce applicable U.S. regulations could be punished, even if its efforts fell short. It was an amazingly permissive statute buried in one of the strictest “obscenity” laws ever passed by Congress.
It was fortunate, then, that before the ink on the CDA had dried, the Supreme Court struck it down. Reno v. American Civil Liberties Union (1997) was the first and most important Supreme Court case to involve the internet. In a unanimous decision, the justices basically laughed the CDA out the door, noting that it massively violated the First Amendment. The only part of the CDA that survived was Section 230. Over the ensuing years, it would be consistently challenged and upheld. With each successful defense, its legal standing grew stronger. Outside of two specific exemptions (federal criminal law and intellectual property), the internet was mostly left to govern itself. As a result, most early corporate censorship—more politely known as “content moderation”—would come not because of government mandate, but to avoid involving government in the first place.
As ever, money was the driver. Over the next decade, questions regarding what constituted permissible online speech centered not on politics or propriety, but on property. Blogger (eventually acquired by Google), for example, was an early self-publishing hub that enabled millions of users to set up websites free of charge. And yet a visitor to Blogger’s home page in 1999 would find no list of rules, only a friendly reminder to properly configure your URL so you could be added to the master blog roll. Some blogs might host racist rants and pornography, the thinking went, but so what? This wasn’t a problem—it was the whole reason services like Blogger existed: to share the panoply of human expression, emotions, and beliefs.
By contrast, violations of intellectual property rights were covered not by the permissive Section 230, but by the much stricter 1998 Digital Millennium Copyright Act (DMCA). This law imposed a maximum prison sentence of five years or a fine of $500,000 for the first offense of posting material for which someone else held a copyright. Fortunately, much like Section 230, the law also included a “safe harbor” provision. If websites promptly responded to a takedown notice filed by the copyright holder—without pausing to consider the merits of the request—they could avoid being sued or jailed.
Ground zero for the copyright battle was YouTube, whose nature made it an irresistible target for hosting copyrighted songs or videos. In 2006, YouTube placed a ten-minute limit on its videos, reasoning that longer clips were likely to be illegally hosted TV shows or movies. A year later, notably following Google’s $1.7 billion acquisition of the company, YouTube introduced its “Content ID” system, which assigned a unique digital fingerprint to tens of millions of copyrighted files. If Content ID spotted a match on YouTube’s servers, the file was automatically flagged for removal. This was the first use of sophisticated, wide-scale automation to control user content on a U.S. website. It was a sign of things to come.
In another sign of things to come, the automated system went too far, disabling videos that contained even an incidental glimpse of copyrighted material. Just a few wayward seconds—like Katy Perry’s “I Kissed a Girl” playing in the background of a video shot in a crowded bar—was enough to nuke a whole clip. In 2008, Republican presidential candidate John McCain complained that his political ads were being automatically removed because they contained brief clips from broadcast news. Digital rights activists gleefully reminded McCain that he’d voted the DMCA into law a decade earlier.
Happily, a small reprieve from copyright laws would arrive later that year, following an epic legal battle between the artist formerly known as Prince and a 13-month-old baby. The baby had been marked as a “copyright violator” after his mother uploaded a video of him pushing a toy stroller and laughing as Prince’s “Let’s Go Crazy” played for precisely twenty seconds in the background. The judicial ruling found, essentially, that the whole thing was ridiculous and that internet users had a right to plead “fair use” before their content was eradicated. YouTube’s copyright-sniffing algorithm was allowed to relax—but not much.
Even as they strengthened their copyright controls, however, emerging social media titans confronted a far more horrifying problem: child pornography. Under Section 230, websites enjoyed broad legal immunity against charges of child abuse or exploitation. But use of a platform by child porn distributors wasn’t just a legal problem; it was a moral imperative, one whose mere mention could sink a company’s reputation. In 2009, Microsoft announced a free service called PhotoDNA. Applying a system much like that of Content ID, PhotoDNA compared each posted image and video with a massive database and instantly flagged any matches. Every major social media platform would eventually adopt the tool, essentially eliminating child pornography from their networks. Today, this top-secret, U.S. government–sanctioned database hosts more than a million instances of child pornography.
Other than addressing copyright infringement and child porn, by the mid-2000s Silicon Valley companies still did little to regulate user speech, clinging to the laissez-faire principles of the first generation of internet pioneers. But as the internet’s population passed 1 billion, that age had clearly ended. Social media platforms were clogged with eager users, including half of all American teenagers. Flush with hormonal drama and anguish, the vast digital commons increasingly resembled a powder keg. All it needed was a spark.
That spark was provided in 2006, when the handsome, athletic, 16-year-old Josh Evans joined Myspace, the then-dominant social network. He liked the bands Rascal Flatts and Nickelback; among his “turn-ons” were tongue piercings and ear nibbling. He’d lived a hard life, born to a single mother who bounced between jobs. Nonetheless, Josh was upbeat. His goal, he confided, was to “meet a great girl.”
Sadly, Josh had one flaw: he wasn’t real. He was a hoax—in the words of journalist Lauren Collins, “an online Frankenstein’s monster.” He was a sockpuppet built to exploit the hopes and vulnerabilities of one teenage girl.
That target was 13-year-old Megan Meier, who—like most teenagers—maintained roller-coaster relationships with her peers. As she entered eighth grade, she fell out with a friend who lived just four houses away. That friend’s mother, 47-year-old Lori Drew, created Josh to spy on Meier, to see if the girl was saying mean things about her daughter. Drew solicited help from a 19-year-old employee of her small business and two other teenagers to run the fake account. Josh began a warm, flirtatious online friendship with Meier. The attractive boy always seemed to know exactly what to say to make Meier happy. Indeed, how could he not? His creator knew Meier well; in a happier time, she had even joined the Drew family on vacations.
The ruse soon turned to tragedy. After Meier got into an angry online argument with other classmates, Josh abruptly turned on her, taking the other kids’ side and peppering her with insults. In shock, Meier fled from the family computer and retreated to her room. When her mother checked on her a short time later, the 13-year-old was dead, hanging from an Old Navy belt. Her bereaved father uncovered the last message sent by Josh: “You’re a shitty person, and the world would be a better place without you in it.”
The story swelled into a major scandal. Drew and her accomplices were tried for conspiracy, convicted, but then acquitted. Their actions were simply too new to have clearly violated any existing laws. That soon changed. As outrage spread and more reports emerged of online harassment, dozens of states enacted “cyberbullying” laws. For the first time, many Americans were forced to reckon with the potentially deadly offline consequences of online actions.
For social media platforms, the death of Megan Meier was also a wake-up call. Myspace had never been in serious legal jeopardy. Indeed, Myspace was technically a victim, listed alongside Megan in the high-profile court case, because Drew had falsely represented herself to the company. But in the court of public opinion, the world’s largest social network faced a public relations disaster. If Myspace had done something, anything, might that young girl still be alive?
For Myspace, Megan’s death was a setback from which it never recovered. For Silicon Valley at large, it was a sign that “terms of service” could no longer be a simple check box to placate jumpy investors. These user agreements had to become a new kind of law, penned by private corporations instead of governments, to administer communities of unprecedented scale. To be effective, these terms would have to be regularly monitored and updated, pulling engineers ever closer to the task of “regulating” free speech. To do otherwise—to leave hundreds of millions of users to say or do whatever they pleased—risked the government jumping in and passing ever more stringent legislation. In other words, to allow truly free speech would be financially ruinous.
Among the companies destined to rule the new social web—Twitter, Google, and Facebook—a shaky consensus emerged. All three banned personal threats or intimidation, soon expanding to include a more general ban on harassment. All but the free-spirited Twitter banned graphic violence and pornography, which included taking a hard-line stance against nudity. These rules seemed simple. They’d prove to be anything but.
For Twitter, staffed by many of the same members of the idealistic team that had founded the original Blogger, the goal was to create a free-flowing platform, a libertarian ideal. “We do not actively monitor and will not censor user content, except in limited circumstances,” declared Twitter’s longstanding mission statement. A Twitter executive proudly described the company as “the free speech wing of the free speech party,” reminding all that it was the place to launch protests and topple dictators. By design, Twitter accounts were anonymous and essentially untraceable. All accounts could speak to all other accounts, unfiltered.
The free speech haven became a perfect platform for rapid news distribution, but also a troll’s paradise. A graphic, personal threat wasn’t allowed, but anything short of that—like telling a Jewish user what would hypothetically happen to them in a second Holocaust—was fair game. The worst fate that could befall a Twitter user was an account ban, but as one neo-Nazi derisively pointed out to us, it took mere seconds to create a new one. As a result, the free speech haven became, in the words of one former employee, a “honeypot for assholes.”
The first case of sustained harassment on Twitter occurred in 2008, as a female tech blogger endured months of insults, threats, and stalking from a network of anonymous accounts. “Twitter is a communication utility, not a mediator of content,” one founder coolly replied, as the backlash grew. For years, despite the paltry protection of its terms of service, Twitter would remain a brutally hostile place for women and nonwhite users. It would take until 2013—amid a massive, sustained harassment campaign against female members of the British Parliament—for Twitter to introduce a way for users to directly report abusive tweets.
A year later came “Gamergate,” an absurd scandal that began over the complaints of an obsessive ex-boyfriend and protests over “ethics in gaming journalism.” It ended with literally millions of abusive tweets hurled at a handful of female video game developers, the effective birth of the alt-right political movement, and an inquest by the United Nations. “Freedom of expression means little as our underlying philosophy if we continue to allow voices to be silenced because they are afraid to speak up,” Twitter’s new general counsel concluded. By the end of 2015, the company’s promise not to censor user content had vanished from its mission statement.
Ironically, YouTube’s more restrictive terms of service (“Don’t cross the line,” it said in big bold letters) led it almost immediately into thorny political questions. The platform banned “unlawful, obscene [or] threatening” videos. But this content proved difficult to define and regulate. The first challenge came in 2007, when Mexican drug cartels flooded the service with music videos that featured the mutilated bodies of their enemies, intended to boost recruitment. YouTube tried to remove the videos as it discovered them. That seemed easy enough. However, the same year, YouTube also deleted the videos of an Egyptian anti-torture activist, whose work necessarily documented torture in order to combat it. Following an angry backlash from human rights groups, those videos were reinstated. In 2008, YouTube removed video of an air strike on a dozen Hamas fighters, whereupon the Israeli Defense Forces complained about the loss of its “exclusive footage.”
But these relatively simple approaches by Twitter (avoid intervention) and YouTube (content bans) hardly registered next to the complex content moderation policies that emerged at Facebook, the company born from a website comparing the hotness of college coeds. From the beginning, Facebook had been gunning for Myspace, so it wanted to avoid as many Myspace-style scandals as possible. Facebook’s internal guidebooks soon came to resemble the constitutional law of a midsized nation. In 2009, its first attempt to codify its “abuse standards” ran to 15,000 words.
Each new rule required more precise, often absurd clarification. It was one thing for Facebook to ban “incitement of violence”; it was quite another to say what that meant. If a user pleaded for someone to shoot the U.S. president, it was a clear incitement and could be deleted. But if a user urged others to “kick a person with red hair,” it was a more general threat and therefore allowable. Indeed, leaked slides of Facebook policy gave a horrifying example of the nuance. The message “Unless you stop bitching I’ll have to cut your tongue out” was permissible because the threats were conditional instead of certain.
Even a seemingly black-and-white rule—like a blanket ban on “nudity and sexual activity”—sowed a minefield of controversy. First came the protests of historians and art critics, who pushed Facebook to allow nudity in photographs of paintings or sculptures but not in digital art, which the classicists considered pornography. Then came the protests of new mothers, furious that their images of breastfeeding were being deleted on the grounds of “obscenity.” They launched a mommy-sourced lobbying campaign, coining their own hashtag, #freethenipple (which was, unsurprisingly, hijacked by porn distributors). These nipple wars led Facebook into years of heated, internal deliberation. Eventually, Facebook’s senior leaders settled on a new policy that permitted portrayals of breastfeeding, but only so long as the nipple was not the principal focus of the image.
The engineers who had built the world’s largest digital platform—which raked in billions of dollars in revenue and shaped news around the world—had neither expected nor wanted to spend hundreds of hours in corporate boardrooms debating the spectrum of nipple visibility. But they did. With great power came great and increasingly expansive responsibilities.
Then came global politics. Originally, a firm could escape onerous censorship requests by arguing that it was a U.S. company subject to U.S. laws. By the early 2010s, this was no longer a realistic defense. These companies had become multinational giants, grappling with regulations across dozens of national jurisdictions. As the scope of Silicon Valley’s ambition became truly international, its commitment to free speech sagged. In 2012, both Blogger (originally marketed as “Push-Button Publishing for the People”) and Twitter (“the free speech wing of the free speech party”) quietly introduced features that allowed governments to submit censorship requests on a per-country basis.
If there was a moment that signified the end of Silicon Valley as an explicitly American institution, it came in 2013, when a young defense contractor named Edward Snowden boarded a Hong Kong–bound plane with tens of thousands of top-secret digitized documents. The “Snowden Files,” which would be broadcast through social media, revealed an expansive U.S. spy operation that harvested the metadata of every major social media platform except Twitter. For primarily U.S.-based engineers, this was an extraordinarily invasive breach of trust. As a result, Google, Facebook, and Twitter began publishing “transparency reports” that detailed the number of censorship and surveillance requests from every nation, including the United States. “After Snowden,” explained Scott Carpenter, director of Google’s internal think tank, “[Google] does not think of itself all the time as an American company, but a global company.”
From this point on, social media platforms would be governed by no rules but their own: a mix of remarkably permissive (regarding threats and images of graphic violence) and remarkably conservative (regarding nudity). In essence, a handful of Silicon Valley engineers were trying to codify and enforce a single set of standards for every nation in the world, all in an attempt to avoid scandal and controversy. As any political scientist could have told them, this effort was doomed to fail.
“Literally, I woke up in a bad mood and decided someone shouldn’t be allowed on the internet. No one should have that power.”
It was August 2017 and Matthew Prince, cofounder and CEO of the Cloudflare web hosting service, had just made a decision he’d spent a decade dreading. Cloudflare had been built to protect websites from cyberattacks, the kind that often happened when someone attracted too much negative attention. Thanks to Cloudflare, dissidents around the world were shielded from unfriendly hackers. But so, too, were the internet’s most abhorrent voices.
For years, Stormfront, a neo-Nazi forum board, had relied on Cloudflare to keep its servers running. Now, in the aftermath of a deadly white nationalist terror attack in Charlottesville, Virginia, Stormfront users were openly celebrating the murder. As outrage against Stormfront and its media outlet, The Daily Stormer, intensified, the anger also targeted Cloudflare, whose technology was keeping the neo-Nazis online. The company meekly explained that it couldn’t revoke Stormfront’s accounts without “censoring the internet”—a position that led Stormfront to brag that Cloudflare was on its side. Seeing this, a furious Prince abruptly reversed course and pulled the plug. He explained his change of heart in an email to his staff:
This was my decision. Our terms of service reserve the right for us to terminate users of our network at our sole discretion. My rationale for making this decision was simple: the people behind the Daily Stormer are assholes and I’d had enough.
Let me be clear: this was an arbitrary decision. It was different than what I’d talked with our senior team about yesterday. I woke up this morning in a bad mood and decided to kick them off the internet . . . It’s important that what we did today not set a precedent.
Saying that something shouldn’t set a precedent doesn’t stop it from doing so. This was a landmark moment. An ostensibly “content-neutral” company had made a decision to destroy content—a decision that was obviously not neutral. And it had happened because a single person at the top had changed his mind. But at least he was being transparent about it.
Prince’s decision echoed the dilemma that increasingly fell to social media’s ruling class. Faced with vocal campaigns to censor or delete speech, the companies could either ignore their users, risking a PR disaster, or comply and be drawn deeper into the political brush. In essence, in avoiding governance, these companies had become governments unto themselves. And like any government, they now grappled with intractable political problems—the kind always destined to leave a portion of their constituents displeased.
Yet they also had little choice. Nations around the world had gradually awoken to the influence that these U.S. social media giants exerted over domestic politics. Between 2012 and 2017, some fifty countries passed laws that restricted the online speech of their citizens. And these weren’t just the wannabe authoritarians discussed in chapter 4; they were also some of the most liberal nations in the world, fearful of terrorism, extremism, or even simply “fake news.” Even in the United States, a new generation of tech-savvy politicians hovered, ready to slap these companies with onerous new government regulations if they didn’t tighten the rules on their own.
No longer was it enough to police copyright infringements, naughty pictures, and the most obvious forms of harassment. Now Silicon Valley firms would be pushed ever closer to the role of traditional media companies, making editorial decisions about which content they would allow on their platforms. Many engineers argued that this was a “slippery slope.” But their founders’ ingenuity and the internet’s exponential growth had placed them in this treacherous terrain. It was now their task to navigate it.
The first and most obvious challenge was terrorism. Very early on, Al Qaeda and its copycats had begun to post their propaganda on YouTube. This included grisly recordings of snipers killing U.S. soldiers in Iraq. Although YouTube technically prohibited graphic violence, it was slow to remove the clips, while the American public was quick to vent its fury.
But the challenge proved even starker with the first internet-inspired terror attacks. The same year YouTube was created, an American-born Islamic cleric named Anwar al-Awlaki became radicalized and moved to Yemen. Charismatic and English-speaking, he began uploading his Quranic lectures to the platform, accumulating millions of views across a 700-video library. Although there was no explicit violence portrayed in the clips of the soft-spoken, bespectacled al-Awlaki, his words promoted violence. And they were incredibly effective, inspiring dozens of deadly attacks around the world, such as the 2009 shooting at Fort Hood, Texas, that claimed thirteen lives.
Moreover, the YouTube algorithm exacerbated the threat by creating a playlist of “related videos” for its viewers. In al-Awlaki’s case, this meant the platform was helpfully steering viewers of his videos to videos by other terrorist propagandists.
By 2011, the U.S. government had had enough, and al-Awlaki was sentenced to death in absentia by an Obama administration legal memo stating that his online propaganda “posed a continuing and imminent threat of violent attack.” Soon after, he was slain by a U.S. drone strike. On YouTube, however, al-Awlaki’s archive became something else: a digital shrine to a martyr. In death, al-Awlaki’s online voice grew even more popular, and the U.S. intelligence community began noticing an uptick in views of his videos that accompanied spikes in terrorist attacks. This illustrated another conundrum. The government had done what it could to silence the “bin Laden of the internet,” but it was up to the engineers at YouTube to determine the terrorist’s future influence. It would take the company another six years, until 2017, to decide to block the videos.
Yet it was Twitter, not YouTube, that became terrorists’ main social media haven. In a horrifying irony, terrorists who wanted to destroy freedom of speech found perfect alignment with Twitter’s original commitment to freedom of speech. The only line a terrorist couldn’t cross was personal harassment. You could tweet, generally, about how all “kuffar” (non-Muslims) deserved a violent death; you just couldn’t tell @hockeyfan123 that you were going to cut off his head. Although many voiced frustration that terrorists were allowed on the platform, Twitter brushed off their complaints. If the NATO coalition could tell its side of the story in Afghanistan, the thinking went, why not the Taliban? For aspiring terror groups, Twitter then became not just the space to connect with followers, but also the perfect platform to build brand recognition among both recruits and Western journalists.
But then came headlines Twitter couldn’t ignore. In 2013, four gunmen stormed Nairobi’s Westgate shopping mall, murdering 67 people and wounding nearly 200 more. The attackers belonged to Al-Shabaab, an East African terror organization whose members had been early and obsessive Twitter adopters. Shabaab applied digital marketing savvy to the attack, pumping out a stream of tweets, press releases, and even exclusive photos (snapped by the gunmen themselves). “#Westgate: a 14-hour standoff relayed in 1400 rounds of bullets and 140 characters of vengeance,” summarized one terrorist’s post. Soon Shabaab became the main source for international journalists writing about the attack—a position the group used to spread misinformation and confuse the situation on the ground even more. Reeling from bad press, Twitter intervened in a way it had been unwilling to do just a few years earlier. It suspended the terrorists’ account. It didn’t matter; Shabaab simply registered new ones.
And then, in 2014, the Islamic State roared onto the global stage, seizing hold of the internet’s imagination like a vise. At its peak, the ISIS propaganda machine would span at least 70,000 Twitter accounts, a chaotic mix of professional media operatives, fanboys, sockpuppets, and bots. As ISIS propaganda seeped across the platform in more than a dozen languages, Twitter executives were caught flat-footed. Their content moderation team simply wasn’t equipped to deal with the wholesale weaponization of their service. This wasn’t just for lack of interest, but also for lack of resources. Every employee hour spent policing the network was an hour not spent growing the network and demonstrating investor value. Was the purpose of the company fighting against propaganda or for profitability?
Meanwhile, public outrage mounted. In 2015, Congress edged as close as it had in a decade to regulating social media companies, drafting a bill that would have required the disclosure of any “terrorist activity” discovered on their platforms (the definition of “terrorist activity” was kept intentionally vague). The same year, then-candidate Donald Trump seemed to endorse the internet censorship and balkanization practiced by authoritarian nations. “We have to talk to [tech CEOs] about, maybe in certain areas, closing that internet up in some ways,” he declared. “Somebody will say, ‘Oh freedom of speech, freedom of speech.’ Those are foolish people.”
Twitter tried to act, but ISIS clung to it like a cancer. Militants developed scripts that automatically regenerated their network when a connection was severed. They made use of Twitter blocklists—originally developed to fight harassment by bunching together and blocking notorious trolls—to hide their online activities from users who hunted them. (ISIS media teams soon added us to this list.) Some accounts were destroyed and resurrected literally hundreds of times, often with a version number (e.g., @TurMedia335). When the rather obvious Twitter handle @IslamicState hit its hundredth iteration, it celebrated by posting an image of a birthday cake. Nonetheless, the growing suspensions changed the once-free terrain for ISIS. “Twitter has become a battlefield!” lamented one ISIS account by mid-2015.
Thanks to diligent volunteer efforts, steady improvements to Twitter’s systems, and unrelenting public pressure, ISIS’s use of the platform gradually declined. In 2017, Twitter announced that its internal systems were detecting 95 percent of “troubling” terrorist accounts on its own and deleting three-quarters of them before they made their first tweet. It was a remarkable achievement—and an extraordinary turnabout from Twitter’s laissez-faire approach of just a few years before.
Although Twitter’s transformation was the most dramatic, the other Silicon Valley giants charted a similar path. In 2016, Google piloted a program that used the advertising space of certain Google searches (e.g., “How do I join ISIS?”) to redirect users to anti-ISIS YouTube videos, carefully curated by a team of Google counter-extremism specialists. It spoke to the seriousness with which Google was starting to treat the problem. Meanwhile, Facebook built a 150-person counterterrorism force to coordinate its response effort, comprised of academics and former intelligence and law enforcement officers.
At the end of 2016, Facebook, Microsoft, Twitter, and Google circled back to where online censorship had begun. Emulating the success of Content ID and PhotoDNA, which had curbed copyright violations and child porn respectively, the companies now applied the same automated technique to terrorist propaganda, jointly launching a database for “violent terrorist imagery.” Just a few years before, they had insisted that such a system was impossible, that the definition of “terrorism” was too subjective to ever be defined by a program. This was another sign of how decisively the political landscape had shifted.
Yet no matter how much these social media companies evolved, there were always outside forces pressuring them to do more. In 2015, Facebook was sued for $1 billion by relatives of Americans who had been killed during a spate of lone wolf terror attacks in Gaza. The tech firm was accused of having “knowingly provided material support” to the terrorists, simply by giving them the means to transmit their propaganda. Around the same time, 20,000 Israelis brought suit against Facebook not just for the violence they’d suffered, but for future violence they feared they might suffer. “Facebook and Twitter have become more powerful today . . . than the [UN] secretary-general, the prime minister of Israel, and the president of the United States,” declared one of the plaintiffs, whose father had been murdered in a Palestinian terror attack. Although the lawsuits were eventually dismissed, each new terror attack prompted further lawsuits by victims. The legal protections granted by Section 230—originally meant to police pornography—had now been pushed to the limit.
Meanwhile, the precedent set by Silicon Valley’s well-publicized purge of ISIS accounts steered it toward other, even more painfully ambiguous political challenges. By 2015, ultranationalists, white supremacists, and anti-immigrant and anti-Islamic bigots had begun to coalesce into the alt-right movement. Feeling emboldened, they increasingly took their hatred into the open.
But they were sly about it. They shrouded their sentiments in memes and coy references; they danced to the very edge of the line without crossing it. The alt-right leader Richard Spencer, for instance, didn’t use his popular (and verified) Twitter profile to directly champion the killing of all Jews and blacks; instead, he simply observed how much nicer things would be if America were made white and pure. The extremists toyed with new ways of targeting people with anti-Semitic harassment. As an example, the last name of someone known or thought to be Jewish would be surrounded by triple parentheses, so that “Smith” became “(((Smith))).” Such tactics made it easier for Gamergate-style hordes to find their targets online and bury them with slurs and abuse. If challenged, they claimed that they were “just trolling.” If their user accounts were threatened, they’d flip to play the victim, claiming that they were being targeted for practicing “free speech.” It represented both a twist on Russian tactics and a deft use of the same language that companies like Twitter had invoked for so many years.
For a time, Google, Facebook, and Twitter essentially threw up their hands and looked the other way. Racism and bigotry were distasteful, the companies readily admitted, but censoring distasteful things wasn’t their job. They also lay within the political spectrum of American politics—at the extreme, to be sure, but gradually becoming mainstream, pushed by their technology. Plus, the tactics of these extremists—winking and nudging, dog-whistling and implying—were often too subtle for any terms of service to adequately address.
But as Silicon Valley cranked up the pressure on terrorists and their supporters, it became easier to contemplate the next step: moving to combat a more general kind of “extremism” that evaded labels, but whose victims—women and ethnic and religious minorities—were easy to name. In mid-2016, Twitter fired the first salvo, kicking the Breitbart writer and far-right provocateur Milo Yiannopoulos out of its service. Having won fame with his race-baiting, Yiannopoulos had finally crossed the line when he organized a campaign of online harassment targeting an African American actress for the crime of daring to star in a Ghostbusters remake.
While Yiannopoulos would insist that he’d been wrongly smeared as a bigot—that he’d “just been trolling”—the evidence suggested otherwise. A year later, when a trove of Yiannopoulos’s files leaked online, it was revealed that he used email passwords like “Kristallnacht” (a November 1938 attack on German Jews in which dozens were murdered) and “LongKnives1290” (a reference to both the Night of the Long Knives, a 1934 Nazi purge that solidified Hitler’s rule, and the year in which Jews were banished from medieval England).
Following a spate of more than 700 hate crimes across the country after the election of Donald Trump in November 2016, pressure began to build on the social media giants to do more about the hate that was not just allowed but empowered by their platforms, especially as it spurred violence. The crackdown started with the long-overdue Twitter suspension of white supremacist leader Richard Spencer. He issued a dramatic rebuttal on YouTube titled “The Knight of Long Knives.” “I am alive physically,” he explained to his followers, “but digitally speaking, there has [sic] been execution squads across the alt-right . . . There is a great purge going on.”
But this digital purge was actually only a time-out. Confident and mobilized in a way that hate groups had not been since the mass KKK rallies of the 1920s, the alt-right used social media to organize a series of “free speech” events around the nation, culminating in that infamous Charlottesville rally. “As you can see, we’re stepping off the internet in a big way,” one white supremacist bragged to a reporter as the air was suffused with neo-Nazi chants. “We have been spreading our memes, we’ve been organizing on the internet, and so now [we’re] coming out.”
Amid the national outcry that followed, the social media giants moved to expand their definition of “hate speech” and banish the worst offenders from their services. Twitter banned the most virulent white supremacist accounts, while Facebook removed pages that explicitly promoted violent white nationalism. Reddit rewrote its terms of service to effectively outlaw neo-Nazi and alt-right communities. White supremacists even found themselves banned from the room-sharing service Airbnb and the dating site OkCupid.
This was a massive shift for an industry barely over a decade old. Since their founding, social media companies had stuck by the belief that their services were essentially a “marketplace of ideas,” one in which those that came to dominate public discourse would naturally be the most virtuous and rational.
But Silicon Valley had lost the faith. Social media no longer seemed a freewheeling platform where the best ideas rose to the surface. Even naïve engineers had begun to recognize that it was a battlefield, one with real-world consequences and on which only the losers played fair. Their politics-free “garden” had nurtured violence and extremism.
The trouble went deeper than the specters of terrorism and far-right extremism, however. Silicon Valley was beginning to awaken to another, more fundamental challenge. This was a growing realization that all the doomsaying about homophily, filter bubbles, and echo chambers had been accurate. In crucial ways, virality did shape reality. And a handful of tech CEOs stood at the controls of this reality-shaping machine—but they hadn’t been working those controls properly.
It was the election of Donald Trump that drove this realization home. Most deeply impacted was Facebook, whose mostly young and progressive employees feared that their work had elevated Trump to high office. Indeed, there was strong evidence that it had. Although Twitter had served as Trump’s treasured microphone, it was Facebook in which Americans had been at their most politically vulnerable, trapped in networks of people who thought just like them and who accorded hundreds of millions of “shares” to stories that were obvious hoaxes. Indeed, the whispers were already beginning that Facebook had been saturated not just with profit-motivated misinformation and “fake news” spun by Macedonian teenagers, but also with a pro-Trump disinformation campaign orchestrated by the Russian government.
As stupefied liberals searched for someone or something to blame, Mark Zuckerberg could see the tidal wave coming. What followed was essentially a corporate version of psychiatrist Elisabeth Kübler-Ross’s five stages of grief: denial, anger, bargaining, depression, and acceptance.
Zuckerberg’s first impulse was to deny. It was “a pretty crazy idea,” he said a few days after the election, that misinformation on his platform had influenced the outcome of anyone’s vote. After his initial denial was met with widespread fury and even a private scolding from President Obama, Zuckerberg shifted gears, penning a series of notes in which he vowed to try harder to counter hoaxes and misinformation on Facebook. At the same time, he tried to reassure users that this was a comparatively small problem. Meanwhile, frustrated Facebook employees began meeting in private to crowdsource solutions of their own. The truth then came out that some at the company had been concerned about rampant misinformation taking place on their platforms during the election, but had been prevented from making any changes for fear of violating Facebook’s “objectivity,” as well as alienating conservative users and legislators.
By mid-2017, Facebook had struck a very different note. In the first report of its kind, Facebook’s security team published “Information Operations and Facebook,” a document explaining how its platform had fallen prey to “subtle and insidious forms of misuse.” In another first, Facebook publicly named its adversary: the government of the Russian Federation. Critics noted, however, that the company had waited a crucial nine months between when its executives knew that a massive campaign of Russian manipulation had occurred on its networks and when it informed its customers and American voters about it.
Reflecting its ability to implement change when so motivated, however, Facebook expanded its cybersecurity efforts beyond regular hacking, turning its focus to the threat of organized disinformation campaigns. Where the company had studiously ignored the effects of disinformation during the 2016 U.S. election, it now cooperated closely with the French and German governments to safeguard their electoral processes, shutting down tens of thousands of suspicious accounts. A year after calling the idea of electoral influence “crazy,” Zuckerberg apologized for having ever said it. And in a very different speech, delivered via Facebook Live, Zuckerberg addressed his 2 billion constituents. “I don’t want anyone to use our tools to undermine democracy,” he said. “That’s not what we stand for.”
This shift was driven in part by a reckoning that their creations had been used and disfigured. Even at the freewheeling Reddit, CEO Steve Huffman spoke of how he realized Russian propaganda had penetrated the site, but removing it would not be enough. “I believe the biggest risk we face as Americans is our own ability to discern reality from nonsense, and this is a burden we all bear.”
Yet much of the impetus for change came in the form it always had—mounting legal and political pressures. In 2017, over the strenuous objections of Silicon Valley lobbyists and free speech advocates, German lawmakers passed a bill that levied fines as high as $57 million on companies that failed to delete “illegal, racist, or slanderous” posts within twenty-four hours. Closer to home, U.S. legislators launched the first major effort to regulate online political advertisements, especially the “dark ads” used by Russian propagandists to spread disinformation and by the Trump campaign to suppress minority voter turnout. It moved to subject them to the same Federal Election Commission disclosure rules that applied to broadcast television. Previously, political advertising on social media—a multibillion-dollar industry—had enjoyed all the same exemptions as skywriting.
For the titans of industry turned regulators of online war, it was an unexpected, unwanted, and often uncomfortable role to play. As Zuckerberg confessed in a 2018 interview, shortly before he was brought to testify before the U.S. Congress, “If you had asked me, when I got started with Facebook, if one of the central things I’d need to work on now is preventing governments from interfering in each other’s elections, there’s no way I thought that’s what I’d be doing, if we talked in 2004 in my dorm room.”
With each step the social media giants took as they waded deeper into political issues—tackling terrorism, extremism, and misinformation—they found themselves ever more bogged down by scandals that arose from the “gray areas” of politics and war. Sometimes, a new initiative to solve one problem might be exploited by a predatory government (Russia had a very different definition of “terrorism” than the United States) or well-meaning reporting systems gamed by trolls. Other times, it might lead to a clueless and costly mistake by a moderator expected to gauge the appropriateness of content from a country they’d never been to, amidst a political context they couldn’t possibly understand.
One illustration of this problem was a Facebook rule, adopted to improve counterterrorism on the platform, that prohibited any positive mention of “violence to resist occupation of an internationally recognized state.” From an engineering standpoint, it was an elegant solution—brief and broad. It was also one that any savvy political observer could have predicted would lead to massive problems. It led to mass deletions of user content from Palestine, Kashmir, and Western Sahara, each a political and cultural powder keg ruled by an occupying power.
These gray areas ran the gamut. A Chinese billionaire, taking refuge in the United States and vowing to reveal corruption among the highest ranks of the Communist Party, found his Facebook profile suspended for sharing someone else’s “personally identifiable information” (which had kind of been the point). In Myanmar, when members of the Rohingya Muslim minority tried to use Facebook to document a government-led ethnic-cleansing campaign against them, some found their posts deleted—for the crime of detailing the very military atrocities they were suffering.
Throughout this messy and inexorable politicization, however, there was one rule that all of Silicon Valley made sure to enforce: the bottom line. The companies that controlled so much of modern life were themselves controlled by shareholders, their decision-making guided by quarterly earnings reports. When a Twitter engineer discovered evidence of massive Russian botnets as far back as 2015, he was told to ignore it. After all, every bot made Twitter look bigger and more popular. “They were more concerned with growth numbers than fake and compromised accounts,” the engineer explained.
When Facebook employees confronted Mark Zuckerberg about then-candidate Trump’s vow to bar all Muslims from entering the United States, he acknowledged that it was indeed hate speech, in violation of Facebook’s policies. Nonetheless, he explained, his hands were tied. To remove the post would cost Facebook conservative users—and valuable business.
It was exactly as observed by writer Upton Sinclair a century earlier: “It is difficult to get a man to understand something when his salary depends on his not understanding it.”
Today, the role of social media firms in public life is one that evades easy description. They are profit-motivated, mostly U.S.-based businesses that manage themselves like global governments. They are charmingly earnest, preaching inclusivity even as they play host to the world’s most divisive forces. They are powerful entities that pretend to be powerless, inescapable political forces that insist they have no interest at all in politics. In essence, they are the mighty playthings of a small number of young adults, who have been given the unenviable task of shaping the nature of society, the economy, and now war and politics. And although the companies and those who lead them have matured an extraordinary amount in just a few years, the challenges they face only grow more complex.
But the most important part of their work is finding the answer to an obvious question—the kind of question that engineers like to hear. Assume that they’ve accepted the scope and complexity of their responsibilities, that they have decided to outlaw an unacceptable behavior and even defined exactly what that behavior looks like. How do they build the systems to stop it? What do those systems look like? Their answers have been to turn to the very same tools that created many of the problems in the first place: the online crowd and pitiless machines.
America Online called them “community leaders,” but this vague corporatese hardly described who they were or what they did. Nevertheless, a time traveler from thirteenth-century Europe would have recognized their role immediately. They were serfs—peasants who worked their feudal lord’s land in return for a cut of the harvest. AOL’s serfs just happened to toil through a dial-up modem. And their lord just happened to be the first true internet giant.
By the mid-1990s, AOL had evolved from a small internet service provider into a sprawling digital empire. For millions of users, AOL was the internet: an online chat service, a constellation of hosted websites (AOL partnered with everyone from CNN to the Library of Congress) and forum boards, and a general internet browser. AOL was both a piece of software and a massive media service, one that eventually reached 26 million subscribers. It marketed itself by carpet-bombing millions of homes with blue CDs emblazoned with the AOL logo, promising “500 Hours Free!” At one time, half of all the CDs produced on earth were used for AOL free trials.
Early in its corporate existence, AOL recognized two truths that every web company would eventually confront. The first was that the internet was a teeming hive of scum and villainy. The second was that there was no way AOL could afford to hire enough employees to police it. Instead, AOL executives stumbled upon a novel solution. Instead of trying to police their sprawling digital commonwealth, why not enlist their most passionate users to do it for them?
And so the AOL Community Leader Program was born. In exchange for free or reduced-price internet access, volunteers agreed to labor for dozens of hours each week to maintain the web communities that made AOL rich, ensuring that they stayed on topic and that porn was kept to a minimum. Given special screen names, or “uniforms,” that filled them with civic pride, they could mute or kick out disruptive users.
As AOL expanded, the program grew more organized and bureaucratic. The Community Leader Program eventually adopted a formal three-month training process. Volunteers had to work a minimum of four hours each week and submit detailed reports of how they’d spent their time. At its peak, the program boasted 14,000 volunteers, including a “youth corps” of 350 teenagers. AOL had effectively doubled its workforce while subsidizing roughly 0.0005 percent of its subscriber base, all while maintaining a degree of plausible deniability if anything went wrong. It seemed to be the best investment AOL ever made.
Predictably, such a criminally good deal was bound for a criminal end. In 1999, two former community leaders sued AOL in a class-action lawsuit, alleging that they’d been employees in a “cyber-sweatshop” and that some were owed as much as $50,000 in back pay. A legal odyssey ensued. In 2005, AOL terminated the Community Leader Program, bestowing a free twelve-month subscription on any remaining volunteers. In 2008, AOL was denied its motion to dismiss the lawsuit. And at last, in 2010—long after AOL had been eclipsed by the likes of Google and Facebook—the company suffered its final indignity, forced to pay its volunteer police force $15 million in back pay.
The rise and fall of AOL’s digital serfs foreshadowed how all big internet companies would come to handle content moderation. If the internet of the mid-1990s had been too vast for paid employees to patrol, it was a mission impossible for the internet of the 2010s and beyond. Especially when social media startups were taking off, it was entirely plausible that there might have been more languages spoken on a platform than total employees at the company.
But as companies begrudgingly accepted more and more content moderation responsibility, the job still needed to get done. Their solution was to split the chore into two parts. The first part was crowdsourced to users (not just volunteers but everyone), who were invited to flag content they didn’t like and prompted to explain why. The second part was outsourced to full-time content moderators, usually contractors based overseas, who could wade through as many as a thousand graphic images and videos each day. Beyond establishing ever-evolving guidelines and reviewing the most difficult cases in-house, the companies were able to keep their direct involvement in content moderation to a minimum. It was a tidy system tacked onto a clever business model. In essence, social media companies relied on their users to produce content; they sold advertising on that content and relied on other users to see that content in order to turn a profit. And if the content was inappropriate, they relied on still other users to find it and start the process of deletion.
When you report something on Facebook, for instance, you’re propelled down a branching set of questions (“Is it a false story?” “Is it pornography?” “Is it just annoying?”) that determine who reviews it and how seriously the report is taken. In this fashion, Facebook users flag more than a million pieces of content each day. The idea of user-based reporting has become so ingrained in the operations of the social media giants that it now carries a certain expectation. When Facebook came under fire in 2017 for allowing the livestreamed murder of a 74-year-old grandfather to remain viewable for more than two hours, it had a ready excuse: nobody had reported it. In effect, it was Facebook’s users—not Facebook—who were at fault.
And then there are the people who sit at the other end of the pipeline, tech laborers who must squint their way through each beheading video, graphic car crash, or scared toddler in a dark room whose suffering has not yet been chronicled and added to Microsoft’s horrifying child abuse database. There are an estimated 150,000 workers in these jobs around the world, most of them subcontractors scattered across India and the Philippines.
Like most outsourcing, it is competitive and decently compensated work, given the pay scales in these locales. Most of it is done by bright young college graduates who would otherwise find themselves underemployed. It takes brains and good judgment to decipher context in just a few seconds, applying all the proper policies and procedures. Thus, the most apt parallel to these jobs isn’t the click farms where laborers endlessly repeat the rote process of SIM card swapping and account creation, but Russia’s sockpuppets and troll factories, which also recruit from the ranks of underemployed, English-speaking college graduates. In a way, the occupations are mirrors of each other. Professional trolls try to make the internet worse. Professional content moderators try to make it a little better.
Unsurprisingly, this work is grueling. It’s obviously unhealthy to sit for eight or more hours a day, consuming an endless stream of all the worst that humanity has to offer. There’s depression and anger, vomiting and crying, even relationship trust issues and reduced libido. In the United States, companies that conduct this work offer regular psychological counseling to counter what they call “compassion fatigue”—a literal exhaustion of the brain’s ability to feel empathy for others in harm’s way. It may not be enough. In 2017, two former Microsoft employees assigned to the Online Safety Team sued their former employer, alleging that they’d developed post-traumatic stress disorder. It was the first lawsuit of its type. One of the plaintiffs described how he’d developed an “internal video screen” of horror that he couldn’t turn off.
Aside from the problems of worker PTSD, this bifurcated system of content moderation is far from perfect. The first reason is that it comes at the cost of resources that might otherwise be plowed into profit generators like new features, marketing, or literally anything else. Accordingly, companies will always view it as a tax on their business model. After all, no startup ever secured a round of funding by trotting out a shiny new gold-plated content moderation system.
The second problem is scale. To paraphrase Winston Churchill, never before has so much, posted by so many, been moderated by so few. When WhatsApp was being used by ISIS to coordinate the first battle for Mosul, the company had just 55 employees for its 900 million users. But even that made it a behemoth. When the newly launched video-hosting startup Vid.Me found itself infested by thousands of ISIS propaganda clips around the same time, the company had a total of just 6 people on staff, none of whom spoke Arabic.
Even these numbers pale in comparison with the true social media giants. Recall from chapter 3 the wealth of data that these services generate. Every minute, Facebook users post 500,000 new comments, 293,000 new statuses, and 450,000 new photos; YouTube users, more than 400 hours of video; and Twitter users, more than 300,000 tweets. Each of these posts is a Sword of Damocles hanging over the company. It can suffer devastating PR disasters if it allows any objectionable piece of content to stand for more than a few minutes before being deleted. But if a company acts rashly, the cries of censorship are liable to come just as fast.
Finally, if social media firms are to police their networks (which, remember, they don’t really want to do), they must contend not just with millions of pieces of content, but also with adversaries who actively seek to thwart and confuse their content moderation systems. Think of the Islamic State’s resilient, regenerating Twitter network, the Russian government’s believable sockpuppets, or the smirking alt-right memes that straddle the line between hateful jokes and raw hate. When Facebook announced in 2017 that it was hiring 250 more people to review advertising on the platform, New York University business professor Scott Galloway rightly described it as “pissing in the ocean.”
Under extraordinary pressure and facing an ever-expanding content moderation queue, the engineers of Silicon Valley have cast far and wide for an answer. Unsurprisingly, they think they’ve found that answer in more technology.
“YOU LOOK LIKE A THING AND I LOVE YOU.”
As a Tinder pickup line, it needed work. But it wasn’t bad for something that hadn’t even been written by a human. All AI specialist Janelle Shane had done was compile a list of existing pickup lines and taught the computer to read them. After that, an artificial brain—a neural network—studied the list and invented a new pickup line all on its own.
Neural networks are a new kind of computing system: a calculating machine that hardly resembles a “machine” at all. Although such networks were theorized as far back as the 1940s, they’ve only matured during this decade as cloud processing has begun to make them practical. Instead of rule-based programming that relies on formal logic (“If A = yes, run process B; if A = no, run process C”), neural networks resemble living brains. They’re composed of millions of artificial neurons, each of which draws connections to thousands of other neurons via “synapses.” Each neuron has its own level of intensity, determined either by the initial input or by synaptic connections received from neurons farther up the stream. In turn, this determines the strength of the signal these neurons send down the stream through their own dependent synapses.
These networks function by means of pattern recognition. They sift through massive amounts of data, spying commonalities and making inferences about what might belong where. With enough neurons, it becomes possible to split the network into multiple “layers,” each discovering a new pattern by starting with the findings of the previous layer. If a neural network is studying pictures, it might start by discovering the concept of “edges,” sorting out all the edges from the non-edges. In turn, the next layer might discover “circles”; the layer after that, “faces”; the layer after that, “noses.” Each layer allows the network to approach a problem with more and more granularity. But each layer also demands exponentially more neurons and computing power.
Neural networks are trained via a process known as “deep learning.” Originally, this process was supervised. A flesh-and-blood human engineer fed the network a mountain of data (10 million images or a library of English literature) and slowly guided the network to find what the engineer was looking for (a “car” or a “compliment”). As the network went to work on its pattern-sorting and the engineer judged its performance and tweaked the synapses, it got a little better each time. Writer Gideon Lewis-Kraus delightfully describes the process as tuning a kind of “giant machine democracy.”
Today, advanced neural networks can function without that human supervision. In 2012, engineers with the Google Brain project published a groundbreaking study that documented how they had fed a nine-layer neural network 10 million different screenshots from random YouTube videos, leaving it to play with the data on its own. As it sifted through the screenshots, the neural network—just like many human YouTube users—developed a fascination with pictures of cats. By discovering and isolating a set of cat-related qualities, it taught itself to be an effective cat detector. “We never told it during the training, ‘This is a cat,’” explained one of the Google engineers. “It basically invented the concept of a cat.”
Of course, the neural network had no idea what a “cat” was, nor did it invent the cat. The machine simply distinguished the pattern of a cat from all “not-cat” patterns. Yet this was really no different from the thought process of a human brain. Nobody is programmed from birth with a set, metaphysical definition of a cat. Instead, we learn a set of catlike qualities that we measure against each thing we perceive. Every time we spot something in the world—say, a dog or a banana—we are running a quick probabilistic calculation to check if the object is a cat.
Feed the network enough voice audio recordings, and it will learn to recognize speech. Feed it the traffic density of a city, and it will tell you where to put the traffic lights. Feed it 100 million Facebook likes and purchase histories, and it will predict, quite accurately, what any one person might want to buy or even whom they might vote for.
In the context of social media, the potential uses for neural networks are as diverse as they are tantalizing. The endless churn of content produced on the internet each day provides a limitless pipeline of data with which to train these increasingly intelligent machines.
Facebook is a fertile testing ground for such neural networks—a fact appreciated by none more than Facebook itself. By 2017, the social media giant had plunged into the field, running more than a million AI experiments each month on a dataset of more than a billion user-uploaded photographs. The system had far surpassed Facebook’s already-uncanny facial recognition algorithm, learning to “see” hundreds of distinct colors, shapes, objects, and even places. It could identify horses, scarves, and the Golden Gate Bridge. It could even find every picture of a particular person wearing a black shirt. If such a system were unleashed on the open internet, it would be like having 10,000 Bellingcats at one’s fingertips.
For the social media giants, an immediate application of this technology is solving their political and business problem—augmenting their overworked human content moderation specialists with neural network–based image recognition and flagging. In late 2017, Google announced that 80 percent of the violent extremist videos uploaded to YouTube had been automatically spotted and removed before a single user had flagged them.
Some at these companies believe the next stage is to “hack harassment,” teaching neural networks to understand the flow of online conversation in order to identify trolls and issue them stern warnings before a human moderator needs to get involved. A Google system intended to detect online abuse—not just profanity, but toxic phrases and veiled hostility—has learned to rate sentences on an “attack scale” of 1 to 100. Its conclusions align with those of human moderators about 90 percent of the time.
Such neural network–based sentiment analysis can be applied not just to individual conversations, but to the combined activity of every social media user on a platform. In 2017, Facebook began testing an algorithm intended to identify users who were depressed and at risk for suicide. It used pattern recognition to monitor user posts, tagging those suspected to include thoughts of suicide and forwarding them to its content moderation teams. A suicidal user could receive words of support and link to psychological resources without any other human having brought the post to Facebook’s attention (or even having seen it). It was a powerful example of a potential good—but also an obvious challenge to online privacy.
Social media companies can also use neural networks to analyze the links that users share. This is now being applied to the thorny problem of misinformation and “fake news.” Multiple engineering startups are training neural networks to fact-check headlines and articles, testing basic statistical claims (“There were x number of illegal immigrants last month”) against an ever-expanding database of facts and figures. Facebook’s chief AI scientist turned many heads when, in the aftermath of the 2016 U.S. election, he noted that it was technically possible to stop viral falsehoods. The only problem, he explained, was in managing the “trade-offs”—finding the right mix of “filtering and censorship and free expression and decency.” In other words, the same thorny political questions that have dogged Silicon Valley from the beginning.
Yet the most important applications of neural networks may be in simulating and replacing the very thing social networks were designed for: us. As we saw earlier, bots pose as humans online, pushing out rote messages. Their more advanced version, chatbots, are algorithms designed to convey the appearance of human intelligence by parroting scripts from a vast database. If a user says something to one of these “dumb” chatbots (“How’s the weather?”), the chatbot will scan all previous instances in which the question appears, choosing a response whose other data points best align with those of the current conversation (if, for instance, the user has previously disclosed that her name is Sally or that she’s from the United States and likes guns). No matter how convincing it is, though, each chatbot is basically reciting lines from a very, very long script.
By contrast, neural network–trained chatbots—also known as machine-driven communications tools, or MADCOMs—have no script at all, just the speech patterns deciphered by studying millions or billions of conversations. Instead of contemplating how MADCOMs might be used, it’s easier to ask what one might not accomplish with intelligent, adaptive algorithms that mirror human speech patterns.
But the development of next-generation MADCOMs also illustrates a flaw inherent in all neural networks: they are only as good as their inputs—and only as moral as their users. In 2016, Microsoft launched Tay, a neural network–powered chatbot that adopted the speech patterns of a teenage girl. Anyone could speak to Tay and contribute to her dataset; she was also given a Twitter account. Trolls swarmed Tay immediately, and she was as happy to learn from them as from anyone else. Tay’s bubbly personality soon veered into racism, sexism, and Holocaust denial. “RACE WAR NOW,” she tweeted, later adding, “Bush did 9/11.” After less than a day, Tay was unceremoniously put to sleep, her fevered artificial brain left to dream of electric frogs.
While the magic of neural networks might stem from their similarity to human brains, this is also one of their drawbacks. Nobody, their creators included, can fully comprehend how they work. When the network gets something wrong, there’s no error log, just the knowledge that, with enough synaptic fiddling, the problem might be fixed. When there’s no way to know if the network is wrong—if it’s making a prediction of the future based on past data—users can either ignore it or take its prognostication at face value. The only way to understand a neural network is to steal a page from neuroscience, monitoring different groups of artificial neurons and testing different patterns to see what stimulates them. Ironically, neuroscientists who conduct similar experiments on human brains (like monitoring the electrical activity produced by each of 10,000 different words) have begun to use neural networks to map and model their results.
The greatest danger of neural networks, therefore, lies in their sheer versatility. Smart though the technology may be, it cares not how it’s used. These networks are no different from a knife or a gun or a bomb—indeed, they’re as double-edged as the internet itself.
Governments of many less-than-free nations salivate at the power of neural networks that can learn millions of faces, flag “questionable” speech, and infer hidden patterns in the accumulated online activity of their citizens. The most obvious candidate is China, whose keyword-filtering and social credit system will benefit greatly from the implementation of such intelligent algorithms. In 2016, Facebook was reported to be developing such a “smart” censorship system in a bid to allow it to expand into the massive Chinese market. This was an ugly echo of how Sun Microsystems and Cisco once conspired to build China’s Great Firewall.
But it doesn’t take an authoritarian state to turn a neural network toward evil ends. Anyone can build and train one using free, open-source tools. An explosion of interest in these systems has led to thousands of new applications. Some might be described as “helpful,” others “strange.” And a few—though developed with the best of intentions—are rightly described as nothing less than “mind-bendingly terrifying.”
We’ve already seen how easy it is for obvious falsehoods (“The world is flat”; “The pizza parlor is a secret underage sex dungeon”) to take hold and spread across the internet. Neural networks are set to massively compound this problem with the creation of what are known as “deep fakes.”
Just as they can study recorded speech to infer meaning, these networks can also study a database of words and sounds to infer the components of speech—pitch, cadence, intonation—and learn to mimic a speaker’s voice almost perfectly. Moreover, the network can use its mastery of a voice to approximate words and phrases that it’s never heard. With a minute’s worth of audio, these systems might make a good approximation of someone’s speech patterns. With a few hours, they are essentially perfect.
One such “speech synthesis” startup, called Lyrebird, shocked the world in 2017 when it released recordings of an eerily accurate, entirely fake conversation between Barack Obama, Hillary Clinton, and Donald Trump. Another company unveiled an editing tool that it described as “Photoshop for audio,” showing how one can tweak or add new bits of speech to an audio file as easily as one might touch up an image.
Neural networks can synthesize not just what we read and hear but also what we see. In 2016, a team of computer and audiovisual scientists demonstrated how, starting with a two-dimensional photograph, they could build a photorealistic, three-dimensional model of someone’s face. They demonstrated it with the late boxing legend Muhammad Ali, transforming a single picture into a hyperrealistic face mask ready to be animated and placed in a virtual world—and able to rewrite the history of what Muhammad Ali did and said when he was alive.
This technology might also be used to alter the present or future. Using an off-the-shelf webcam, another team of scientists captured the “facial identity” of a test subject: the proportions of their features and the movement patterns of their mouth, brows, and jawline. Then they captured the facial identity of a different person in a prerecorded video, such as Arnold Schwarzenegger sitting for an interview or George W. Bush giving a speech. After that, they merged the two facial identities via “deformation transfer,” translating movements of the first face into proportionate movements by the second. Essentially, the test subject could use their own face to control the expressions of the person onscreen, all in real time. If the petite female in front of the webcam opened her mouth, so did the faux Arnold Schwarzenegger. If the middle-aged guy with spiky hair and a goatee mouthed words in rapid succession and raised an eyebrow, so did the photorealistic George W. Bush. As the researchers themselves noted, “These results are hard to distinguish from reality, and it often goes unnoticed that the content is not real.”
Neural networks can also be used to create deep fakes that aren’t copies at all. Rather than just study images to learn the names of different objects, these networks can learn how to produce new, never-before-seen versions of the objects in question. They are called “generative networks.” In 2017, computer scientists unveiled a generative network that could create photorealistic synthetic images on demand, all with only a keyword. Ask for “volcano,” and you got fiery eruptions as well as serene, dormant mountains—wholly familiar-seeming landscapes that had no earthly counterparts. Another system created synthetic celebrities—faces of people who didn’t exist, but whom real humans would likely view as being Hollywood stars.
Using such technology, users will eventually be able to conjure a convincing likeness of any scene or person they or the AI can imagine. Because the image will be truly original, it will be impossible to identify the forgery via many of the old methods of detection. And generative networks can do the same thing with video. They have produced eerie, looping clips of a “beach,” a “baby,” or even “golf.” They’ve also learned how to take a static image (a man on a field; a train in the station) and generate a short video of a predictive future (the man walking away; the train departing). In this way, the figures in old black-and-white photographs may one day be brought to life, and events that never took place may nonetheless be presented online as real occurrences, documented with compelling video evidence.
And finally, there are the MADCOMs. The inherent promise of such technology—an AI that is essentially indistinguishable from a human operator—also sets it up for terrible misuse. Today, it remains possible for a savvy internet user to distinguish “real” people from automated botnets and even many sockpuppets (the Russified English helped us spot a few). Soon enough, even this uncertain state of affairs may be recalled fondly as the “good old days”—the last time it was possible to have some confidence that another social media user was a flesh-and-blood human being instead of a manipulative machine. Give a Twitter botnet to a MADCOM and the network might be able to distort the algorithmic prominence of a topic without anyone noticing, simply by creating realistic conversations among its many fake component selves. MADCOMs won’t just drive news cycles, but will also trick and manipulate the people reacting to them. They may even grant interviews to unwitting reporters.
Feed a MADCOM enough arguments and it will never repeat itself. Feed it enough information about a target population—such as the hundreds of billions of data points that reside in a voter database like Project Alamo—and it can spin a personalized narrative for every resident in a country. The network never sleeps, and it’s always learning. In the midst of a crisis, it will invariably be the first to respond, commanding disproportionate attention and guiding the social media narrative in whichever direction best suites its human owners’ hidden ends. Matthew Chessen, a senior technology policy advisor at the U.S. State Department, doesn’t mince words about the inevitable MADCOM ascendancy. It will “determine the fate of the internet, our society, and our democracy,” he writes. No longer will humans be reliably in charge of the machines. Instead, as machines steer our ideas and culture in an automated, evolutionary process that we no longer understand, they will “start programming us.”
Combine all these pernicious applications of neural networks—mimicked voices, stolen faces, real-time audiovisual editing, artificial image and video generation, and MADCOM manipulation—and it’s tough to shake the conclusion that humanity is teetering at the edge of a cliff. The information conflicts that shape politics and war alike are fought today by clever humans using viral engineering. The LikeWars of tomorrow will be fought by highly intelligent, inscrutable algorithms that will speak convincingly of things that never happened, producing “proof” that doesn’t really exist. They’ll seed falsehoods across the social media landscape with an intensity and volume that will make the current state of affairs look quaint.
Aviv Ovadya, chief technologist at the Center for Social Media Responsibility at the University of Michigan, has described this looming threat in stark, simple terms. “We are so screwed it’s beyond what most of us can imagine,” he said. “And depending how far you look into the future, it just gets worse.”
For generations, science fiction writers have been obsessed with the prospect of an AI Armageddon: a Terminator-style takeover in which the robots scour puny human cities, flamethrowers and beam cannons at the ready. Yet the more likely takeover will take place on social media. If machines come to manipulate all we see and how we think online, they’ll already control the world. Having won their most important conquest—the human mind—the machines may never need to revolt at all.
And yet, just as in the Terminator movies, if humans are to be spared from this encroaching, invisible robot invasion, their likely savior will be found in other machines. Recent breakthroughs in neural network training hint at what will drive machine evolution to the next level, but also save us from algorithms that seek to manipulate us: an AI survival of the fittest.
Newer, more advanced forms of deep learning involve the use of “generative adversarial networks.” In this type of system, two neural networks are paired off against each other in a potentially endless sparring match. The first network strains to create something that seems real—an image, a video, a human conversation—while the second network struggles to determine if it’s fake. At the end of each match, the networks receive their results and adjust to get just a little bit better. Although this process teaches networks to produce increasingly accurate forgeries, it also leaves open the potential for networks to get better and better at detecting fakes.
This all boils down to one important, extremely sci-fi question. If both networks are gifted with ever-improving calibration and processing power, which one—the “good” AI or the “bad” AI—will more often beat the other?
In the answer quite possibly lies not just the fate of content moderation policy, but also of future wars and elections, as well as democracy, civilization, and objective reality. Within a decade, Facebook, Google, Twitter, and every other internet company of scale will use neural networks to police their platforms. Dirty pictures, state-sponsored botnets, terrorist propaganda, and sophisticated disinformation campaigns will be hunted by machine intelligences that dwarf any now in existence. But they will be battled by other machine intelligences that seek to obfuscate and evade, disorient and mislead. And caught in the middle will be us—all of us—part of a conflict that we definitely started but whose dynamics we will soon scarcely understand.
It is a bizarre, science-fiction-seeming future. But for something that began with an SF-lovers email thread, it also seems strangely appropriate.