Understanding Commercial
Content Moderation
As we have seen, commercial content moderation is the organized practice of screening user-generated content posted to internet sites, social media, and other online outlets. The activity of reviewing user-generated content may take place before the material is submitted for inclusion or distribution on a site, or it may take place after material has already been uploaded. In particular, content screening may be triggered as a result of complaints about material from site moderators or other site administrators, from external parties (for example, companies alleging misappropriation of material they own), or from other users who are disturbed or concerned by what they have seen and then trigger mechanisms on a site, an action called the “flagging” of content, to prompt a review by professional moderators.1
Commercial content moderation is an essential practice in the production cycle for commercial websites, social media platforms, and media properties that solicit content from users as a part of their online presence. For the companies employing this practice, moderation and screening are crucial steps that protect their corporate or platform brand (by enforcing user adherence to site guidelines or rules), ensure compliance with the laws and statutes governing their operations, and contribute positively to maintaining an audience of users willing to upload and view content on their sites.
Yet the process to handle this content is often catch-as-catch-can. On many highly trafficked sites, the amount of user-generated content submitted is staggering—and growing. Issues of scale aside, the complex process of sorting user-uploaded material into either the acceptable or the rejected pile is far beyond the capabilities of software or algorithms alone. Not just a technical or computational issue, the long history of theoretical challenges in sorting and classifying information has been well documented by scholars like Geoffrey Bowker and Susan Leigh Star and is the starting point for the challenge of content moderation of this scope and volume.2 It is where the nature of the content (that is, what it is or what it depicts), its intent (that is, what it is meant to do when consumed or circulated), its unintended consequences (that is, what else it might do, beyond its first-order intent), and its meaning (which can be highly specific, culturally, regionally, or otherwise) all intersect. From there, it must then be evaluated against “the rules,” both of the platform or local context of the ecosystem (its social norms, expectations, tolerances, and mores) and of the larger, open world and its social, cultural, commercial, and legal regimes and mandates.
Some content lends itself to partial batch processing or other types of machine-automated filtering, particularly when that material is extant and already resident in databases of known bad material. But given the complexity of these processes and the many issues that must be weighed and balanced all at once, the vast majority of social media content uploaded by its users requires human intervention for it to be appropriately screened—particularly where video or images are involved. Human screeners are called upon to employ an array of high-level cognitive functions and cultural competencies to make decisions about their appropriateness for a site or platform. Importantly, for most mainstream social media platforms that rely on user participation, the norm is to allow all uploads without significant pre-screening, meaning that, with a few exceptions, any adjudication or decision making about the appropriateness of a particular video, image, or text post will likely come after it has already been on the platform for some period of time. This paradigm was a business decision on the part of the social media companies themselves, and certainly not a foregone conclusion based on technological necessity or other factors. It is one that users have now typically come to expect.
Professional moderators must be experts in matters of taste of the site’s presumed audience and have cultural knowledge about the location where the platform is based and the platform’s audience. Both the headquarters of the tech platform and its audience may be very far removed, geographically and culturally, from the location where workers are viewing and moderating the user-generated content. Moderators must have linguistic competency in the language of the content (which may be a learned or second language for the screener), be steeped in the relevant laws governing the site’s country of origin, and have expert knowledge of user guidelines and other incredibly detailed platform-level specifics concerning what is and is not allowed.
Some material uploaded by users is inappropriate based on general site guidelines, or under the law. In such cases, moderators can make quick and easy decisions about it. They may cull their queues of user-generated material by engaging computational tools like automated text searches for banned words (to moderate text-based comment sections), “skin filters” to determine if a large portion of an image or video shows bare flesh (suggesting, but not always indicating, pornography), or tools designed to match, flag, or remove copyrighted material.3 Yet even when aided by these mechanisms that may speed up or even automate aspects of the moderation process, the vast amount of uploaded user content has typically required human evaluation and review, particularly when flagged by other users.
Screenshot of a portion of a YouTube page describing material prohibited for users to upload, offering insights into the kinds of video that CCM employees must screen out. These guidelines have morphed over the years. As shown here in July 2017, they appealed to users’ “common sense” to not “cross the line.” YouTube states that it draws the line at graphic and violent content, and also copyright infringement. The former is allowed, the guidelines note, if necessary for a “news or documentary context.”
In the case of images and video, machine-automated detection remains an incredibly complex computational problem. The subfield known as “computer vision”—computer recognition of images and objects—is an ongoing area of research that presents numerous technological challenges, frequently making it computationally and financially infeasible to implement across the board and at scale in many content moderation environments.4 These tasks therefore fall to humans who work through queues of digital material, employing their own decision-making process and, if necessary, intervention.
Whether undertaken through computational or human means, or likely through some combination thereof, the organized, professional full-time work of moderating user-generated content has been a relatively unknown and frequently not-fully-disclosed aspect of participation in social media and websites and services that rely on user-generated content. Many online social media sites and other platforms that trade in user-uploaded material consider the detailed internal-facing specifics of their moderation practices and their policies to be proprietary information. In their view, full disclosure of the exact nature of their policies could lead to unscrupulous users attempting to game the rules, or potentially give business competitors an edge by revealing practices or processes considered secret. The maintaining of these policies as proprietary and secret has also allowed the firms to escape scrutiny and public review of these policies from their users, civil society advocates, and regulators alike. Indeed, content moderators are frequently required to sign nondisclosure agreements (NDAs) about the nature of their work.
For companies whose profits depend on material generated by users, there is, however, at least one other factor that encourages them to keep their content moderation practices veiled. They have viewed their need for large-scale, industrialized content moderation as an unpleasant necessity, that, were it known and fully understood, would have the potential to reveal an underside of sites as mechanisms of distribution for users wishing to circulate unpleasant, distasteful, and disturbing material—not the image most mainstream platforms are eager to cultivate. These platforms therefore engage commercial content moderators to perform tasks that oscillate between the mind-numbingly repetitive and mundane to exposure to images and material that can be violent, disturbing, and, at worst, psychologically damaging. Further, under current regimes that have been largely dictated by the social media firms requiring commercial content moderation, these tasks are demanded of workers who are frequently relatively low-status and low-wage in relation to others in the tech industry and sometimes even in the same building.
The reality of the working conditions of commercial content moderators and their necessity to the infrastructure of online participation offers an unpleasant behind-the-curtain vision that few social media companies or other platforms and sites soliciting user-uploaded content care to openly discuss. While reporting on an early news story focusing on the practice of industrial-scale content moderation, Rebecca Hersher of NPR’s All Things Considered was denied access to both Microsoft and Google employees. Instead, for her report from 2013, “Laboring in the Shadows to Keep the Web Free of Child Porn,” a Microsoft spokesperson, in a significant understatement, described content moderation work as “a yucky job.”5
To understand the practice of commercial content moderation, it is necessary to gain a picture of how, where, and by whom the work of moderation takes place. Content moderation or screening of social media is part of a production cycle of analytical work that, according to Michael Hardt and Antonio Negri, “creates immaterial products, such as knowledge, information, communication, a relationship, or an emotional response.”6 This social media production is, in turn, facilitated by digital networks in a globally connected environment, arrangements that media studies scholar Michelle Rodino-Colocino describes as “technomadic.”7
One of my first key discoveries as I researched industrial-scale moderation work for social media was that the work itself is fractured organizationally and geographically, a fact that was not well known or understood, even by those in the industry itself. Commercial content moderators, I realized, labor under a number of different regimes, employment statuses, and workplace conditions around the world—often by design. Frequently, they are deployed far away from the physical locations where the content they moderate is created, and also at great distance from the hosting sites of the platforms for which the material is destined. Their work titles often differ, ranging from “content moderator,” to “screener,” to “community manager,” to a host of other labels that may sometimes be euphemistic, fanciful, or imply little about their expected job activities—let alone their relation to others doing similar work. In fact, even the workers themselves have difficulty recognizing one another by job title alone. Indeed, because the job titles they toil under are so multitudinous and the job sites so dispersed, it is difficult for researchers, reporters, and labor advocates to locate and identify commercial content moderation workers. Journalists like Julia Angwin, Olivia Solon, Davey Alba, and others, however, have worked tirelessly to overcome these barriers to examine the corporate practices of companies like Facebook and Google, and their networks of labor relationships for moderation tasks.8
Rather than occurring in a monolithic and easily definable workplace or as a set of work practices, commercial content moderation crosses several industrial sectors. It manifests in a variety of worksites and under working conditions that vary with regard to status and worker relationship to the platform and firm ultimately in need of the gatekeeping practice. The Caleris workers in Iowa, for example, were laboring in what was, for all intents and purposes, a third-party call center. As I tracked job listings and labor markets for digital or information work, I found that workers doing what appeared to be commercial content moderation were distributed globally in places like India, Ireland, and the Philippines. Later in my research I encountered workers employed on site at the headquarters of a major tech firm I refer to as “MegaTech.” And I discovered job postings on microlabor platforms like Amazon Mechanical Turk looking to hire people for what they called “content review tasks,” yet another term for commercial content moderation. The landscape was varied and complex, requiring theoretical and geographic mapping.
To get an idea of these intricate structures and relationships, I have created a taxonomy of the locations and employment conditions that are typical for most content moderation workers (Table 1). I define four basic types of work situations for professional moderators: in-house, boutiques, call centers, and microlabor websites.
The first type is “in-house,” a complex concept in the context of online content screening. It can denote a range of employment statuses and relationships to the company and platform or site for which the moderation is being performed. It might connote full-time employment with the company/site or other, less permanent arrangements. Some examples might include workers who perform their moderation labor at the company’s physical site (such as a headquarters or company-owned property) alongside other workers, but possess a status other than full-time employee of that company. Their employment status may take forms like “temporary” or “contract,” where workers are hired by the platform requiring content moderation, but for a delimited period and with no guaranteed future employment within the company after the contract has been fulfilled.
Another “in-house” arrangement might have workers working on site at the company in need of moderation, but with these workers being hired, managed, and paid by a third-party contracting company or companies. This contracting arrangement is fairly common in the information technology (IT) industry for low- or entry-level positions, or for fixed-term positions, which is how many content moderation jobs are classified.9 The primary characteristic of “in-house” moderators is that they are physically located at the platform or company for which the screened content is ultimately destined, although very little can be surmised about their employment status with that company based on their physical presence on site alone. On-site commercial content moderators are likely to be found in divisions with names like “Trust & Safety” or “Community Operations” at the firms that have them, including Facebook, Twitter, YouTube, and Snap, among many others.
Second, a “boutique” arrangement refers to specialized firms that offer social media brand management, generally, or content moderation specifically, for other firms. These businesses are specialists in managing and staying abreast of many areas of a company’s online presence, and they are typically engaged by client companies that are not digital media or technology companies themselves. Although these clients do not themselves specialize in social media and are not primarily engaged in encouraging user-generated content creation or distribution as a fundamental facet of their business or operations, they employ features (for example, comments sections, user picture uploads) that solicit such content to encourage customer or citizen engagement and loyalty. This content requires monitoring, screening, and moderation.
In recent years, companies like the U.K.-based eModeration (now known as “The Social Element”) or the highly stylized California-based ModSquad have provided these specialized services. Such companies may manage brand identity across numerous online platforms and sites, including the company’s own website, Twitter accounts, Facebook pages, and so on.10 In many cases, not only do they moderate and curate engagement with user-generated content on behalf of other companies, they may even engage in “community management” practices: seeding content by posting comments, tweeting, or otherwise attempting to engage consumers in positive conversations or interactions regarding brands or products of the companies for which they work. OnlineExperts, a boutique firm described in detail in Chapter 4, is an exemplar of this arrangement.
Third, I define the “call center” environment as third-party companies offering a suite of services (frequently referred to as business process outsourcing, or BPO), among which moderation of user-generated content is but one service, and is often secondary to call center and other customer-support operations. These companies have the benefit of already being highly technologically enhanced, a necessary feature in order to handle globalized high-volume phone or data traffic; the addition of content moderation work can therefore be passed along to them with ease and efficiency. These centers are globally dispersed, with many call centers now based in the Philippines, although many others are located in India, Bangladesh, and other regions of the world (including the United States, Ireland, and Italy).11 These centers rely on a multilingual and multiculturally competent workforce that works on site at the call center to respond to the labor needs of a global marketplace, often on a 24/7 cycle. In call center environments catering to Western business needs that are themselves located outside the West, issues of cultural and linguistic authenticity can serve as mechanisms for firms to differentiate themselves and to suggest a higher level of service. Workers in these environments are asked to perform linguistic and cultural norms that match those of the clientele, but which are often at odds with their own local cultural and linguistic self-expression. This disconnect can result in the translation of a job well done into measures of how good an employee may be at cultural and linguistic passing.12
The fourth and final type I identify is “microlabor websites.” Content moderation as microlabor digital piecework has been perhaps an obvious site of expansion for globally networked microlabor websites. These digital labor marketplaces connect parties seeking task completion and those seeking employment on a per-task or per-job basis. Sites such as Upwork (formerly oDesk.com) and Amazon Mechanical Turk allow workers to solicit knowledge work tasks that can be performed as long as the worker has access to a computer and connection to the internet in order to bid on the job and then perform it.13 In the case of Amazon Mechanical Turk, owned and operated by Amazon.com, the site focuses on discrete job units (“human intelligence tasks,” or HITs, in the platform’s parlance) that are computationally difficult to achieve, such as content screening and moderation. Its website describes its philosophy for prospective workers:
Amazon Mechanical Turk is based on the idea that there are still many things that human beings can do much more effectively than computers, such as identifying objects in a photo or video, performing data de-duplication, transcribing audio recordings, or researching data details. Traditionally, tasks like this have been accomplished by hiring a large temporary workforce (which is time consuming, expensive, and difficult to scale) or have gone undone.14
Microlabor sites represent the most disconnected and disjointed of all the arrangements under which content moderators labor. Additionally, accountability to the worker—for the job tasks, work environment, rate of pay, and so on—is the lowest of all the forms described here. Under the microlabor arrangement, workers can be the most globally dispersed and isolated of all content moderation workers. Their tasks are broken out to their smallest component parts, such that one view of one image may constitute one job, with payment being as low as one cent (U.S. $0.01) for applying a commercial content moderation decision to that image.15 Fundamentally precarious, workers have no official status with any company, receive no salary or hourly remuneration for their work and no benefits, and are incentivized to do as many tasks as possible to accumulate payment.16 It is, in effect, digital piecework as a form of gig work, which has been written about extensively by scholars like Lily Irani, Ayhan Aytes, Six Silberman, and Jamie Woodcock, among others.17
Commercial content moderation work can be found on microlabor task sites such as Amazon Mechanical Turk. In this case, from October 2011, the assignments shown offer a caveat, “Worker discretion is advised.” The tasks are viewing and making decisions about potentially “offensive” material or images that “may contain adult content” for a “reward,” or payment, of one cent each.
During the course of my research, I found that companies requiring content moderation services customarily develop hybrid strategies to engage workers through dimensions of the above taxonomy and across numerous labor sites worldwide. They may, for example, maintain a suite of in-house contract workers at the company headquarters who are retained and paid by an outsourcing firm. The in-house team may, in turn, collaborate with teams working at call centers elsewhere in the world, such as in Manila or Gurgaon, who respond to a subset of cases requiring review. A company may hire a boutique firm to manage all aspects of its social media brand identity, including review content uploaded by its users, or it may turn, instead, to microlabor sites like Upwork or Amazon Mechanical Turk to deal with the overwhelming inflow of user-generated content they receive, on a piece-by-piece basis. The largest firms may employ any or all of these techniques and forms of labor to meet their user-generated content processing needs while carefully watching their bottom line, attending to the massive volume of material requiring review, and attempting to reduce both the labor and technology costs associated with content moderation practices whenever possible.
The accuracy of my findings about the hybridity of techniques that platforms deploy to engage workers was underscored when I began interviewing U.S.-based commercial content moderation workers. One research participant who worked as an in-house moderator at MegaTech in Silicon Valley said, “When our shift ends, the team in India comes online.” When I asked him if the workers were MegaTech employees, as he saw himself, or contractors, he replied, “Contractors.” So, while the MegaTech workers in California fell into the “in-house” category as on-site contractors nonetheless paid by a third-party contracting firm, when contrasted with Indian counterparts, the in-house MegaTech mods saw them as distinctly other: squarely in the “call center” designation and unequivocally contractors at a far remove from MegaTech. This is only one example of how such hybrid strategies are employed to meet the demand of 24/7 moderation at massive scale for sites and platforms with a global user base, while reducing costs for the firms that need such moderation coverage and introducing complex layers of labor stratification and reporting structures. Additionally, social media platforms may turn to these strategies because so much of the work necessitates linguistic and cultural competencies present in abundance only in particular regions, so the locations of the third-party companies offering moderation services, and the skills of their workforce, matter greatly and become a competitive selling point for their services in the marketplace.
Whatever the specific arrangements and strategies implemented to get the work of content moderation done, there tend to be shared characteristics at all levels and among all sites of user-generated content moderation and screening. Screening and moderation tasks are customarily performed by contracting with semipermanent or temporary workers in low-wage, low-status environments. Commercial content moderation itself is both a global industry and set of practices. The work is often performed well removed from the site—and often the country or region—for which the content under review originated or was initially destined. Finally, content moderation of user-generated content fits precisely into new forms of technology-dependent knowledge labor envisioned, initially and optimistically, as having the potential for providing a better quality of work and life, a higher standard of living, and more leisure time to the broad strata of workers engaged in it. In contrast to how that labor environment was envisioned, however, commercial content moderation is work that calls for very large labor forces of not particularly high technical skill engaged in repetitive, unpleasant work. As social media firms’ quest for workers willing to perform digital piecework proceeds on a global scale, the remuneration for their labor is offered at ever lower rates. In this way, online content moderation and screening may represent a dystopian, technologically enhanced race to the bottom. The location of the Mechanical Turk worker population, for example, has largely shifted to India, and many who “Turk” as a part- or full-time job receive two dollars per hour or less.18
In whatever context a worker might find herself performing commercial content moderation tasks, the likelihood is that the work is not a job for which she has trained, prepared, or studied. Before responding to an ad, being recruited or approached by a contracting agency, or being contacted via other, often indirect or circuitous means, most commercial content moderators have never heard of this type of work. Even following their initial job interviews, they may still not fully understand the nature of the work, which, until relatively recently, did not even exist.
A Brief History of Knowledge Labor in the “Post-Industrial Society”
In 1973, against the backdrop of the Cold War, the Vietnam era, and the advances in science and technology that fueled and were stoked by both, sociologist Daniel Bell published his foundational work The Coming of Post-Industrial Society. In it, Bell put forth his vision of a socioeconomic paradigm shift so profound as to represent a “change in the social framework of Western society,” and specifically in its economic structure, technological capability, and occupational systems.19 Characteristics of these shifts, according to Bell, included a movement from a commodity, or goods-oriented, economy to one focused on the service sector; the rise of the technical class engaging in specialized, scientific, or other technical work (for example, data analysis, engineering); and the increased importance and predominance of technological innovation. In concert, Bell prophesied, these shifts would result in a service-based, technology-driven economy, and its stock-in-trade would be production and analysis of new knowledge—the desired output in the post-industrial society and a key component of the socioeconomic arrangements to come.
These shifts, already under way at the time of Bell’s writing and with additional shifts “forecasted” (his preferred term), represented an evolutionary leap from the status quo of the structures in place since the great industrialization of the nineteenth century that had endured throughout most of the twentieth century. In the United States, the twentieth century’s sites of production were typically factories producing tangible goods en masse. Organizationally, the factories were often arranged in a vertical hierarchy of management and workers in Taylorist tradition, all of whom were long-term employees, generally working on a manufacturing production line. This type of organization and production is usually described as “Fordist,” related to the automobile manufacturer’s innovations in assembly line production and other efficiencies in production and labor. Accordingly, Bell’s description of and prediction for the post-industrial era concluded that the industrial era’s commitment to innovation in routinization, mechanization, and other efficiencies of scientific management practices would be carried forward and enhanced by the forecasted socioeconomic shifts.
The imagined result of these shifts would see workers move out of the factories and into offices, working more efficiently but ostensibly for fewer hours, with an enhanced quality of work life and an increase in leisure time. American society would then surge even further to the fore in industries of the future that would be based not on tangible goods manufacturing but rather on scientific and technological innovation. As knowledge production would increasingly become the most important output of workers, the movement from an industrial society, based on mass production of tangible goods, to a post-industrial society, based on the production and, thereby, the commodification of information, would increase.
Indeed, from the 1970s through the mid- to late 1980s, many (but not all) of Bell’s predictions had come true: mass production of goods had declined in the United States, while new sectors of innovation in science and technology experienced unprecedented growth. Silicon Valley, for example, experienced its first major period of prominence during this time. By 1976, future tech entrepreneur Marc Porat began his influential dissertation by asserting, “We are now an information economy. Over half our wages and nearly half of our GNP [gross national product] originate with the production, processing and distribution of goods and services,” and by as early as 1967, “over half of all labor income [was] earned by workers whose tasks [were] predominantly informational.”20
Meanwhile, other scholars began to deepen critical analyses of contemporary socioeconomic organization and information and communication technology (ICT) development, many critiquing Bell’s largely optimistic view.21 Sociologist Manuel Castells developed his theory of a “network society,” an information-driven economy characterized by the compression of time and space into a “space of flows” and organization and labor practices reconstituted into flexible, reconfigurable, and dynamic structures that more closely resemble interconnected nodes than did the top-down hierarchies of the factories and plants of the industrial era.22 Such organization, enhanced by the data-driven computational power of global digital connectivity, transcended geospatial boundaries into global networking arrangements and was no longer limited to a traditional workday. Instead, it could function across time zones and around the clock.
Yet this new networked configuration of the post-industrial society did not benefit all. On the contrary, the very characteristics that made it so appealing to some served to reinforce social inequities experienced by many, and to create new ones, too, particularly in arrangements that seemed to benefit the private sector at the expense of the public sphere. Castells, for example, critiqued the post-industrial network society on this basis, cautioning against the new inequities that arose from the geospatial reconfigurations produced by compressed time and space. He described this as a “Fourth World,” a new space of interconnected locales around the world whose commonality and connection was predicated not on geographic proximity or a historic trajectory of underdevelopment (due to colonization and its resource extraction, for example, as was the case for many so-called Third World countries), but due to their shared exclusion from the network society.23 Concern about the exclusion of people from the globally networked world culminated in the identification of the “digital divide” phenomenon in the United States, with an influential study by the National Telecommunications and Information Administration in 1995.24 Research regarding the nature and character of the informational divide continued throughout the 1990s and into the 2000s, with efforts being made in economically diverse areas of the world to mitigate its damaging effects.25 Political economists of communication such as Herbert Schiller and Dan Schiller proposed that the “digital divide” was simply an expression of a larger socioeconomic divide: capitalism’s schisms as perpetuated by the intertwined lack of access to information and computer technology and a stranglehold by a relative few large corporations over their control.26
Indeed, the characteristics of the world economy, too, had begun to significantly shift at the same time as the changes described by Bell, Castells, and others took hold. Markets moved across traditional state and geographic borders in a great global expansion. Because financial information and transactions could more swiftly traverse the globe, respond to changes in markets on a worldwide scale, and be analyzed and reconfigured in increasingly complex, machine-aided ways, governments in major Western countries such as the United States and the United Kingdom increasingly relied on erstwhile fringe economic policies of supply-side economics to advocate and gain a reduction in regulatory and other barriers to market entry and expansion, and the data- and information-driven financial service sector experienced large-scale growth in such areas as investment banking, debt servicing and lending, and speculation.27
These sectors made great gains throughout the past forty years, and relied heavily on the features of post-industrial and network society configurations to realize their gains. Yet these sectors’ trade in intangible goods and the riches made in that trade were hardly distributed equally; rather, they benefited only a very few, and in very few places around the globe. The digitization of financial markets, in particular, has resulted in what legal scholar Frank Pasquale describes as “the black box society,” with its inner workings remaining impenetrable to most and thus increasingly impervious to regulation.28 These same digital markets have also proved extremely fragile and vulnerable to manipulation, particularly in the past few years, which have seen entire stock markets and other intangible “financial products” greatly lose value or collapse altogether.
The New Nature of Work: Knowledge Labor in the Digital Era
The reorganization of labor into new forms of flexible, distributed global practices has been one of the primary characteristics of the post-industrial/network/knowledge society era since it was first theorized and noted—heralded, by some, as potential freedom from the assembly floor and critiqued by others as perhaps different from, but no better than, past arrangements.29 These reorganizations are predicated on features such as the global flow of networked information, the proliferation of a distributed workplace organizational model, and an emphasis on immaterial analytical forms of labor.
Yet in some cases, the status and conditions for labor in the post-industrial era may, in fact, be even worse than in the past. Due to the acceleration and compression of time and space, and the global reach of networks that connect labor markets with workers, there has been in many sectors a redistribution of work to its smallest reducible parts and to the lowest possible bids for that work on the world marketplace, as we have seen in the case of Amazon Mechanical Turk’s HITs: so-called microtasks and other forms of app-driven gig labor that scholars have demonstrated have deeply racialized and gender-based characteristics of exploitation.30 In most cases, these functions, and their concomitant exploitative characteristics, are technologically enhanced by computerization and digitization of that work. What, then, does knowledge labor, and labor under post-industrialism, look like?
Among the many insights offered in his book focused on the post-industrial workplace organization of Indian computer programmers, Virtual Migration, A. Aneesh provides the following description: “Dominating forms of labor are concerned less and less with manipulating and altering physical objects; rather, programming allows what is in effect a liquefaction of labor, by converting different forms of work . . . into code that can flow online [so that] . . . in a marriage of code and capital, labor increasingly moves in [a] code-based transnational space.”31 Theorist Tiziana Terranova underscores the shift from the production of discrete material goods to a paradigm of networked knowledge work that “is about the extraction of value out of continuous, updateable work, and . . . is extremely labor intensive. It is not enough to produce a good Web site, you need to update it continuously to maintain interest in it and fight off obsolescence. Furthermore, you need updateable equipment (the general intellect is always an assemblage of humans and their machines), in its turn propelled by the intense collective labor of programmers, designers, and workers.”32 The work characteristic of this new epoch is therefore cyclical, symbiotic, and self-perpetuating. Further, as Terranova articulates, a fundamental shift has occurred from the industrial to the post-industrial age whereby the material fabrication of machines supports knowledge labor as the product, rather than knowledge being used in the service of building of machines for the manufacturing of physical objects or other products. Likewise, entire secondary immaterial industries and practices, such as commercial content moderation, have developed in support of that knowledge-labor product. Labor forces from around the globe have been drafted to fulfill this new work need.
Other phenomena have arisen, too, out of the digitization of work practices and the primacy of culture-making knowledge labor, such as the rise of play or leisure activities that actually constitute work and, in many cases, may be performed without traditional remuneration to the worker (for example, editing entries in Wikipedia, quality-assurance testing for games, user-tagging, and other crowd-sourced activities), and the tendency for more and more knowledge work to resemble, or be made to resemble, activities of leisure or play.33 In some cases, this could include more historical examples of online moderation, or contemporary examples that follow those models, such as volunteer-driven community moderation on sites like Reddit.
Critical digital media scholar Christian Fuchs offers a further analysis of knowledge work and workers that reveals nuance and stratification between “direct” and “indirect” workers. According to Fuchs, “Direct knowledge workers (either employed as wage labour in firms or outsourced, self-employed labour) . . . produce knowledge goods and services that are sold as commodities on the market (e.g., software, data, statistics, expertise, consultancy, advertisements, media content, films, music, etc.) and indirect knowledge workers . . . produce and reproduce the social conditions of the existence of capital and wage labour such as education, social relationships, affects, communication, sex, housework, common knowledge in everyday life, natural resources, nurture, care, etc.”34 This division maps, in many cases, to differences in socioeconomic valuations of that work, as demonstrated by wage and status differentiation. Fuchs views immaterial knowledge work, whether performed through direct or indirect production, as a fundamental characteristic of globalized capitalism, describing the labor as what “produces and distributes information, communication, social relationships, affects, and information and communication technologies.”35
Commercial content moderation of user-generated content in social media may fit within the bifurcation of knowledge labor as direct or indirect, as described by Fuchs. Or perhaps it constitutes a third, hybrid form of immaterial knowledge labor. This type of labor straddles or bridges the two, requiring specialized cultural, social, and linguistic capital normally associated with direct production of knowledge commodities, for workers to make their judgments based on rules, social norms, and taste (characteristics that seem to fall into the “direct” category), and yet does not produce categorically new knowledge, goods, or services. Rather, commercial content moderation is an act of gatekeeping and curation; simply, albeit importantly, passing along material created by others—a function of replication that would fall into practices labeled by Fuchs as “indirect.” Content moderation labor is, in this way, an act of information processing, appraisal, assessment, and evaluation with similarities to other labor practices reliant upon such skills but seldom recognized as such or considered in the same echelon as work requiring expertise.
These theoretical frameworks for knowledge labor underscore many of the greatest shifts away from past labor practices and paradigms, and into both a product of and an actor in the new socioeconomic configurations of the knowledge-based network society. Yet labor in the knowledge economy does not represent a complete rupture from practices of the previous era; rather, it retains key elements from the industrial age that are sustained and enhanced by network-enabled and -enabling technologies and practices. Ursula Huws, for example, notes the following about traditional call centers (a comparable site for low-status immaterial labor): “In many ways, they fit the model ‘post-industrial’ workplaces: the work is white-collar, requiring considerable amounts of knowledge to perform; it relies crucially on information and communications technology. . . . Yet it exhibits many of the features commonly supposed to epitomise ‘industrial’ Fordist production, including Taylorist management and a work-pace determined by machines and their programmes.”36
According to definitions developed by these and other theorists, commercial content moderation workers fit the description of knowledge workers, an artifact of the technologies of the digital network economy, although their labor differs from the high-status, high-wage, creative-class knowledge workers of the late 1990s and early 2000s.37 Digital knowledge work represents a clear shift away from the agricultural manual labor and manufacturing and industrial production that characterized much of American economic and labor activity until the late twentieth century. Knowledge labor relies instead on a worker’s cultural capital and ability to engage with information and communication technology. Increasingly, it relies, too, on locations other than those of the content-hosting entities providing large, cheap pools of laborers (that is, traditional “outsourcing”), culturally and linguistically fluent in Western culture and in American English, who can be digitally networked anywhere in the world, as is the case with the employees of Caleris, the midwestern U.S. call center profiled in the New York Times article that first caught my eye.
The Globalized Knowledge Workforce: “Outsourcing” in Rhetoric and Practice
Knowledge labor is not the only kind of work that has sprung up in the digital era. On the contrary, the ICT-reliant network economies are also highly dependent on a great deal of heavy industry and manufacturing to meet their equipment and infrastructure needs. Yet such industrial production is frequently located, through processes and structures of globalization, around the world (typically in the Global South) in such a way to best take advantage of lenient and permissive environmental and labor laws and practices, location of natural resource extraction, and cheaper labor, while simultaneously rendering that production invisible to the socioeconomically elite of the world.38
Such arrangements allow the post-industrial era to retain the mythology of its name, an era that, in the West, imagines an evolution beyond the need for coal-belching factories of the nineteenth century or the mind-numbing rote tasks of the twentieth-century assembly line, while engaging in equally environmentally damaging manufacturing activities. Silicon Valley–based labor activist Raj Jayadev has commented on this peculiar collective myopia, saying: “A profound characteristic the popular psyche has accepted about the Information Age is the presumption that technology is produced by some sort of divine intervention so advanced that it requires no actual assembly or manufacturing, the very same features our predecessors in the Industrial Era found so essential. Yet every computer, printer, and technological wizardry in-between bought at the local Radio Shack is birthed in what is usually a very inglorious assembly line production site.”39 This myopia extends, too, to the end of life of these products, frequently hidden from Western consumers via disposal at electronics waste sites in China, the Philippines, India, Ghana, and elsewhere in the Global South.40
Likewise, human intervention and immaterial labor is indeed a key, and yet equally cloaked, part of the production chain in sites that rely on user-generated uploaded content requiring screening. Commercial content moderators serve an integral role in making decisions that affect the outcome of what content will be made available on a destination site. The content screeners also view large amounts of material that never makes it long-term to the site or platform it was intended for, as they deem it unfit based on site guidelines, legal prohibition, or matters of taste—their labor literally remaining unseen to anyone who visits the site.
In many cases, great geospatial reconfigurations have taken place to facilitate this hidden manufacturing, the material and immaterial labor underpinning the knowledge economy, as Aihwa Ong, David Harvey, Dan Schiller, and others have demonstrated. These reconfigurations frequently take the form of “special industrial zones” or “special economic zones,” particularly in East Asia, where terms are favorable for transnational corporations to base major manufacturing operations, or for local corporations working on contract to such international concerns.41 The countries that host them and the companies that take advantage of them treat these zones as having different terms of governance than traditional sovereign nation-states. As such, these zones can attract transnational business interests by offering tax exemptions and other sweetheart economic terms that may also include relaxed labor laws or other incentives that leave workers and other citizens at a deficit.
In this sense, the phenomenon of outsourcing might mean something more complex than simply shifting worksites from one nation-state or geographic location to another. In China, for example, its own internal migration of 150 million workers from primarily rural areas to the manufacturing centers concentrated in its special industrial zones is greater than the transnational migration throughout the rest of the world combined.42 In this way, post-industrial labor configurations have troubled notions of nation-states, border crossing, migration, race, and identity. Highly skilled and well-educated workers from the Global South nevertheless find themselves forced to go abroad for employment, such as the Indian programmers in Berlin profiled in Sareeta Amrute’s Encoding Race, Encoding Class—racialized, largely segregated from local German communities and their permission to reside in the country tied precariously to their status as temporary knowledge workers.43 Meanwhile, Western workers find themselves “outsourced” in their own countries of origin, and compared directly to as well as competing directly with counterparts they likely consider a racialized Other halfway around the globe.
Consider the case of the Iowa-based Caleris, where midwestern values were what was on sale. In 2010, Caleris was a call center company offering a suite of business process outsourcing services to potential clients, including content moderation of material uploaded by users. Headquartered in West Ames, Iowa, its worksites at the time were located in several rural, formerly largely agricultural areas of the state. The catchphrase of the business touted its location and the specialized cultural sensibility of its White, as depicted on its website, midwestern American workers through its xenophobia-tinged slogan, “Outsource to Iowa—not India.” This phrase was coupled with images featuring bucolic cornfields and iconic red barns and silos invoking Iowa’s rich agricultural history, serving as both enticement and reassurance to would-be customers about the cultural orientation, economic status, and political context of the company and its employees. The veracity of the scene was not in question on the Caleris site, but such family farms had by that point mostly given way to large-scale corporate agribusiness throughout the state. Nearby Ames, Iowa, was the site of the Farm Aid benefit concert in 1993, designed to raise awareness and money for family farmers subject to foreclosure.44 Perhaps just one or two generations ago, the Caleris employees of rural Iowa would have been laboring in fields of the kind invoked through the imagery on its splash screen.
By 2010, Caleris employees at its four call centers in formerly agrarian Iowa existed in a space of outsourcing, competing in a global marketplace in which its cultural capital was invoked as a means to give the company an edge. Indeed, Caleris co-founder Sheldon Ohringer noted in a corporate profile that the low cost of labor and of living in rural Iowa, coupled with a perceived “lack of regional accents” and an intangible set of values on the part of the Iowa-based employees, made the company’s commercial content moderation services highly appealing to potential clients. “Iowa, often viewed as the heart of midwestern values, has served as the perfect place to draw talented and hard-working individuals whose value systems tend to align with America’s view of what is appropriate when it comes to user-generated content,” Ohringer said in a press release, favorably comparing his employees with those of direct competitors to his business in countries such as India.45
From 2010 to at least 2013, the Caleris home page highlighted its location in Iowa, “not India,” as a selling point for its contract commercial content moderation and call center services. Caleris went on to change its tagline to “Premium customer support from America’s heartland” before the firm rebranded as Aureon in 2016.
Geospatial, economic, and political reconfigurations of intra- and international outsourcing and direct competition across the globe have been brought to bear in the context of knowledge work and, specifically, in online commercial content moderation. Although it has only one-tenth the population of India, the Philippines has now surpassed that country in terms of the actual number of workers in call centers—frequently the site of moderation of user-generated content for social media companies headquartered on the other side of the globe.46
Many of these global call center firms solicit for online content moderation clients in Western markets and in English. MicroSourcing, for example, based in the Philippines, appeals to a primarily U.S.-based audience with its reference to and rather positive spin on the Philippines’ long history of colonization and cultural domination from the West, specifically and most recently by the United States. On its website, the company touts its employees’ excellent command of English and their immersion in Western popular culture (“thanks to their excellent English writing and communication skills . . . and their understanding of Western slang and figures of speech”) as a selling point for their services, available on a 24/7/365 basis. Elsewhere on the site, MicroSourcing offers a “Virtual Captives” service for firms wishing to develop and rapidly deploy an offshore team.47
The practice of user-generated content moderation at companies like Caleris and MicroSourcing, both of which are call-center-style business process outsourcing firms, poses profound questions about the nature of digital and knowledge labor in the post-industrial era, the nature of the commercialized internet, and the phenomenon of globalization and the socioeconomic and political structures that drive it. Caleris’s own slogan, for example, “Outsource to Iowa, not India,” identifies the contracted labor it undertakes as a form of outsourcing, suggesting the character and nature of the labor performed—contractual, piecework, precarious, networkable, and so on—as a key shared characteristic with other firms engaged in these practices locally. And although it seeks to differentiate itself from India with this slogan, the effect is a rhetorical closing of distance that puts Iowa closer to India in terms of labor pool than a neighboring midwestern American state such as Illinois, Wisconsin, or Nebraska. Those do not register in proximity to Iowa in the map of the world that Caleris has redrawn.
MicroSourcing, a multi-service business process outsourcing firm based in the Philippines, touts its employees’ excellent command of English and their cultural immersion in Western popular culture as a selling point for their content moderation services. The webpage asserts that Filipino people possess innate qualities making them particularly adept at supplying moderation services for North American clientele, including a “great eye for detail.”
For Caleris, even though the worksites and workers remain inside the geographic boundaries of the United States, the company itself sees the service it provides to U.S. companies as one of outsourcing. Therefore, outsourcing is not just a geographic concept denoted by physical-world spatial characteristics or constraints, but something else: a set of labor processes and configurations, and a designation of an available labor pool, as opposed to simply location. It is a type of work, a wage level, a class of workers whose peripheralization is enabled and enhanced by digitization and the internet’s “space of flows”—its cleavage with constraints of chronological time and geophysical space, wherever in the world they may be.48
At the same time, both the rhetoric and the practice of outsourcing rely on a transnational bridging of low-wage, low-status, and contractual labor. Under this logic it therefore becomes reasonable for a company such as Caleris to first identify its direct competition as not another midwestern state with similar socioeconomic, political, and cultural characteristics, but India—and to then differentiate itself from India, the Other, by selling its services on the basis of Caleris’s cultural intangibles that are as ostensibly White, rural, and American as the red barn and midwestern farm depicted on its home page.
Given the financial, public relations, and political implications and impact a viral video or other highly circulated, highly viewed content can have, the tasks performed by these workers are far from inconsequential. What can be stated unequivocally is that human intervention and immaterial labor are indeed a key, and yet obfuscated, part of the production chain in online sites that rely upon user-generated uploaded content to populate and draw in their producers/users/consumers. Content moderators, whose labor and even mere existence are so frequently hidden from view, nevertheless serve an integral role in making decisions that affect the outcome of what content will be made available on a destination site. While the moderators’ work may not be as physically demanding or dangerous as that of those workers whose labor goes into IT hardware manufacturing, it indeed is often disregarded, unmentioned, or unacknowledged—and has its own potential for psychological damage, in terms of the nature of the material to which the moderators may be exposed.
Against the backdrop of a much publicized and, ultimately, troubled initial public offering, Facebook made a smaller splash in the news in 2012, when facts about its user-generated content moderation practices came to light via the online news and entertainment website Gawker. Several digital pieceworkers performing user-generated content screening through a microlabor site contacted Gawker staff journalist Adrian Chen. The workers, who reviewed user-generated content for oDesk (now Upwork) on behalf of Facebook, provided Chen with leaked internal oDesk documents describing Facebook’s content screening standards and practices that highlighted disturbing aspects of what the workers were required to see.
Chen’s subsequent story, “Inside Facebook’s Outsourced Anti-Porn and Gore Brigade, Where ‘Camel Toes’ Are More Offensive Than Crushed Heads,” was remarkable in a number of ways: first, he wrote about real-world examples shared with him by the workers, most of whom were no longer working for Facebook via oDesk as digital pieceworkers, and many of whom were located outside the United States (primarily in the Global South).49 The workers’ accounts gave concrete examples of the kinds of trauma-inducing material they were exposed to while being paid digital piecework–level wages that seemed disproportionate to the psychological hazards of their work.
Second, the workers provided Chen with internal documents from oDesk used for training and quality control by the content screeners. This type of material is generally not available for public view and is considered proprietary information by both the companies providing commercial content moderation and those that contract for moderation of user-generated content destined for their sites. But protecting this information as trade secrets obscures other issues, allowing a company to legally maintain ambiguity about its screening and censoring practices, and giving in to a larger propensity toward a logic of opacity that “can lead users to have wildly different interpretations of the user experience on the same site . . . [and that] renders the machinery of content moderation opaque.”50 More general “user guideline”–style statements give plenty of room in which to operate when making subjective content screening decisions. Chen’s piece also pointed out the strange hierarchy of user-generated material as subject to social media firms’ particular internal logic, and oDesk’s protocols for adjudicating it by the screeners.
While other media seized on the sensationalistic aspects of the story, just below the surface were even more compelling facts.51 For example, the internal oDesk documents provide a great deal of insight into the kinds of material that the low-paid contract laborers were expected to see.52 Images and videos of animal abuse and mutilation, child abuse (physical and sexual), gore, disturbing racist imagery, and so on occurred so often that there were specialized protocols devoted to handling them—keeping them from making it to the site if they were not yet there, and removing them if they were.
Not long after the news about Facebook moderation via oDesk broke, Facebook released a confusing infographic ostensibly designed to shed light on the cryptic route that reported content takes through the company’s circuit of screening. According to the company, content flagged as inappropriate, for any one of myriad reasons, makes its way to “staffers in several offices around the world to handle the millions of user reports it receives every week about everything from spam to threats of violence.”53 Reasons cited in the infographic that may result in material being reported include content that is sexually explicit, involves harm to self or others, depicts graphic violence, contains hate speech, and so on.
What was missing, however, from Facebook’s infographic and accompanying statement was any suggestion of how much content is routed through this circuit, and how significant a problem routinely addressing inappropriate user-generated content tends to be. Critically lacking was any discussion of the status or the working conditions of the “staffers . . . around the world” who contend with this distressing material as a major function of their job. The leak did confirm that Facebook at least at that time had employed microlabor companies such as oDesk/Upwork, and potentially others, to conduct these moderation and review practices. The workers engage in digital piecework offered on microwork sites and are therefore afforded no protections or benefits whatsoever. Further, their physical and geographical isolation from their co-workers means they cannot support or commiserate with one another about the content they view as a condition of their work, a key coping mechanism for content moderation workers.
In this way, Facebook benefited from the lack of accountability that comes with outsourcing; that is, introducing secondary and tertiary contracting firms into the cycle of production—a fact that is critically absent from the infographic. Workers engaged as moderators through digital piecework sites are isolated, with few (if any) options for connecting—for emotional support as well as for labor organizing—with other workers in similar conditions, and without any real connection to the original worksites from which the content emanates.
Another little-publicized aspect of outsourcing is that, while the microwork sites and the major corporations that engage them may tout the ability to draw on expertise from a global labor marketplace, in practice these temporary work relationships result in lost payroll-tax revenue for countries such as the United States when labor is outsourced from it, and in significant increases in these kinds of labor pools in Greece and Spain, countries devastated by economic crisis and crippling “austerity” measures.54 Indeed, the worldwide marketplace for digital piecework allows for bargain-basement rates that, by design, drive the value of the labor down to the lowest global bidder. The connection between economic crisis in a region and an increase in the availability of competent labor that is exceedingly cheap cannot be lost in this analysis.
Why Commercial Content Moderation Matters
As a central and mission-critical activity in the workflow of online digital media production, commercial content moderation is little known, frequently low-wage/low-status, and generally outsourced. Content moderation ensures brand protection, adherence to terms-of-use statements, site guidelines, and legal regimes (for example, copyright, law enforcement). It is a key part of the production chain of commercial sites and social media platforms, yet companies often distance themselves from this work and dislike publicity about their moderation practices.
But it also complicates, troubles, and directly contradicts notions of the internet as a free-speech zone. It introduces the existence of human decision-makers unknown to and unseen by the vast majority of end users, who are nevertheless critical in the production chain of social media decision-making. Their invisible presence disrupts comfortable and commonplace notions predicated on the one-to-one relationship of user-to-platform. It paints a disturbing view of an unpleasant work task that the existence of social media and the commercial, regulated internet, in general, necessitate.
Certainly, early theorists of the post-industrial age envisioned the morphing of the industrial twentieth century into the post-industrial twenty-first century as ripe with possibility for greater flexibility, mobility, higher-status work, and more leisure time for American workers, who would shift to using their analytical and technical acumen developed from their production abilities and migrate to a knowledge-based economy of high-status, high-wage work. Yet, the vision promulgated by Bell and the technological deterministic futurists who came in his wake has increasingly ceded to a different reality. Rather than elevating the workers of the world, twenty-first-century configurations of labor are undergoing a globalized race to the bottom in search of ever cheaper, faster, and more human and material resources to compete in the globalized, 24/7 networked marketplace.
Finally, although there has been a great emphasis and primacy placed on knowledge work under post-industrial social arrangements, heavy industry and manufacturing of all sorts are still alive and critically important, even to the knowledge economy, suggesting a shift rather than supersession. The post-industrial labor economy has demanded great geospatial rearrangements and migrations of people, whose “flexibility” is often synonymous with “instability,” “precarity,” or “marginality,” as scholars like Jack Linchuan Qiu have shown.55 To paraphrase Tiziana Terranova, labor under the knowledge economy may just not be as fun as it has been made out to be.56 In order to contend with what is wrong with labor under the current socioeconomic structures, with the purpose of bettering it, we must continue to unveil the realities of labor and other costs (environmental, for example) of the knowledge and digital economies, on a global scale and in all strata.57