Chapter 3 A New Advertising Food Chain

Starting from zero in 2002, in two years Google made $2.08 billion from the advertising that appeared next to its search engine results.1 Its rocketing “paid search” business marked the beginning of a new chapter in advertising history and had ripple effects throughout the online and off-line media economy. To understand why marketers would get so excited about search-engine advertising, consider that for decades they had been fixated on what they call the “purchase funnel,” or “consumer decision journey”²—the multistage trip that people take when they make a purchase, from awareness of product choices to action on those choices. Google's innovations meant that, for the first time, advertisers could reach out to huge numbers of individuals as they took consumer decision journeys online.

Marketers hoped to connect those individuals conducting Google searches with products that mirror the interests expressed by the search terms, using two approaches: search-engine optimization (SEO) and paid search. The first aims to exploit so-called organic (nonpaid-for) search results—in other words, the list of websites that appear when you type keywords into the search box. In response to those search terms, Google's formulas determine which sites will appear on the list and, depending on a site's relevance, how near the top. Although many of the relevance criteria are secret, two core values that guide a site's appearance and position on Google's search results are well known: the extent of the site's connection to the search terms and its reputation with respect to those topics. Google's signal contribution to search engines was to define a site with a high reputation as one that has many other sites linking to it. A site that reflects the search terms and has many other high-reputation sites linking to it will appear higher on the organic search results compared with sites with fewer such links. Under SEO, marketers attempt to optimize their position on the organic search results by crafting sites sensitive to Google's rules. For example, to give its site the best chance to appear prominently in the search results when an individual is looking for information about high-definition televisions, Toshiba would want to make sure that its website contains the sort of information about its high-definition TV products that would lead many other sites to link to it. Achieving this goal is difficult, as Toshiba is one of many major high-definition television manufacturers, and none of these companies can control its ranking in Google's organic search results. All these manufacturers are constantly jockeying for better placement, and Google adds to the challenge by continually changing its search formulas.

Google's paid-search advertising system is another way to reach prospective customers, and under this process companies are better able—though still not guaranteed—to ensure that their ads appear. This process involves an auction, as companies bid on search words—perhaps thousands of them—to win the right to have its ads appear prominently alongside the organic search results. For Toshiba the terms they might bid on could be as generic as HD-TV and as specific as Toshiba LED. To increase its chances, Toshiba's bid might also even include competitors’ names, such as Samsung, Vizio, and Panasonic. Toshiba's digital-advertising agency would present its bids electronically to Google, and the search engine would evaluate it along with those it receives from other companies based on a complex and changing combination of factors, such as the relevance of the product to the firm's bidding words, the bid amount, and the degree of success (measured via clicks) that other such Toshiba ads have already had on Google. Google's paid-search ads have appeared in different places on the search page over the years. Currently Google ranks its auction winners and arrays them immediately above the organic search findings (usually three ads) as well as to the right (up to eight ads). Google's basic business model, though, has remained the same: every time someone clicks on one of these ads, the advertiser pays Google the agreed-upon amount.

This pay-per-click principle was not new when Google's founders, Larry Page and Sergei Brin, launched their paid-search business sometime around 2002. This approach was the same that had intrigued but then frustrated websites and advertisers, including Procter and Gamble, for its low yield. But the Google crew was confident in the advertising model, which built off the approach Web entrepreneur Bill Gross had used for his search engine GoTo. Later called Overture and ultimately bought by Yahoo!, it used a pay-per-click system with click charges set by auction. Google's savvy innovation was instituting an algorithm for choosing auction winners that considered not just the bid price but also the relevance of the ads according to the number of clicks that they or ads like them had registered in previous campaigns. Page and Brin reasoned that awarding auctions for ads that were also based on click records would both benefit Google's revenues and encourage marketers to create ads relevant to users, according to this new definition of relevance. The pair also diverged from the competition by having the search engine carry text ads only. With many users entering Google via slow dial-up connections, Page and Brin worried that ads with pictures would unduly slow the loading of the search results. They also wanted to emphasize their goal of creating the most relevant search experience. They therefore insisted that the ads be no longer than a couple of lines, be informative, and carry no more than ninety-five characters. Initially a vertical line separated the noncommercial (organic) search results on the left side of the page from paid search findings on the right side.3

It worked. JP Morgan estimated that by 2008 Google had a 73 percent share of search revenues in a $22-billion global search economy.⁴ Many small advertisers aiming to sell products directly off of the click liked the ability to put small amounts of money into keywords that reflected their products, and they accepted the proposition that a click on their ad meant someone at least was paying attention. Although common keywords were cheap in the beginning, Google's increasing traffic as a search engine and expert ability to match search words with ads led to soaring revenues. Research in 2003 by the comScore internet research company for the Internet Advertising Bureau (IAB) found that sponsored-search listings in April and May of that year generated click-through rates of over 16 percent, more than four times greater than those for organic search listings, based on the financial-services and travel-industry ads studied. Over the next few years, the click-through rate plummeted (by 2010 it was less than 1 percent), but the actual number of clicks (and so the payout) remained huge because of the enormous number of searches that internet visitors made at Google.5

Although the pay-per-click metric telegraphed Google as a direct-response vehicle, major marketers got in on the act despite the limiting nature of the ads. For example, they saw the additional utility of integrating search campaigns with brand advertising on traditional media. In 2005, General Motors’ Cadillac division tied its search campaign to its television ads during the Super Bowl, concluding that the two thirty-second TV spots accounted for a 170 percent increase in clicks on the car's search site ads over the previous week. Cadillac planned to increase its search site ad spending 1,000 percent in 2006.⁶

Reports of the Cadillac and similar experiences in the trade publications and at industry meetings came as unwelcome news for website executives. Search site advertising was yet another development that challenged online publishers’ usefulness as an advertising vehicle. Media buyers agreed with Google's claim that consumers’ keywords were insights into their direct interests. Google's pitch that clicking on an ad was a significant indicator of individual consumers’ interest fit exactly with advertisers’ increasing insistence on direct measurability. Moreover, media buyers reasoned that the sometimes-high prices they occasionally had to bid for individual words (“car,” for example) were actually cheaper than the pay-per-click model might appear, because even the text ad had display value, and they didn't pay for that.⁷

By 2010 search site advertising was earning about half of all Web revenue. Not all of this amount would flow to non-search-engine websites if Google and the other search engines didn't exist, but the increasing popularity of search by brand-oriented advertisers such as General Motors did suggest that there was direct competition between publishing and search-engine sites. True, organic listings might lead searchers to websites, and sometimes this “side-door” traffic could substantially boost visitors and thus ad revenue. But for most websites it was an uneven and unpredictable way to make a living. The dilemma for Web publishers—companies outside of search-engine sites that serve up and distribute content—didn't stop there. As they worked furiously to attract advertisers in order to win some of the other 50 percent of the media buyers’ spending with new forms of display advertising, they turned for help to new types of organizations—online ad networks, data providers, and data exchanges. These companies promised to help them boost revenues by presenting advertisers with yet more information about the individuals in their audiences. What emerged quickly, though, was a new advertising food chain in which publishers—particularly traditional publishers that spent a lot of money on original content—fell dangerously toward the bottom.

As usual, the media-buying developments contained seeds of irony. Publishers were key supporters of the new system. They hoped media buyers would see the utility of their Web pages for drawing target audiences. Despite providing more and more data about their visitors in the hope of gaining revenue clout, and despite the spread of broadband, which could accommodate the videos the P&Gs of the world had said they wanted, marketers and their affiliates kept pushing Web publishers further down the ladder. Neither the advertising trade press nor industry meetings paid attention to the possibility that weakening certain kinds of publishers might harm the role they play as sources of culture and democratic argumentation. Nor did executives engage in debates about whether particular actions might positively or negatively affect diversity and quality of entertainment, news, and information. That the positions of these publishers were not considered stands to reason: in the roiling, competitive digital advertising system, companies struggled to prove efficient return on investment. How we get our politics, where we seek our identity, how we frame ourselves and our relationship to the world—these simply were not part of the business agenda.

In fact, by the decade's end many media buyers didn't even consider websites their focus. The internet industry's laserlike attention to audience data had led to technology that allowed agencies to “buy” (that is, purchase the right to reach) individuals, automatically and in real time, on whatever page they landed. The intrinsic value of any particular website threatened to erode. This decoupling of audiences from context marked a historic shift in deciding what content to support.

At the end of 2009, 92 percent of online households (which represented 72 percent of total U.S. households) possessed some kind of high-speed connection—and the percentage was growing.8 A result was the increasing number of rich-media and video ads. Created with the help of companies such as Eyeblaster and EyeWonder, the commercial messages could dive directly in front of site visitors with screen-size presentations as the requested page loaded, elastic boxes that spread out when a person moused over them, and TV-style commercials embedded in page banners and joined to the beginning, middle, or end of site videos. The display ads and videos allowed for high-profile image advertising that aims to cultivate awareness of a product and its personality as well as clicks to the product website to learn more. It's the kind of presentation that Denis Beausejour of P&G wanted back in 1998. It's also the kind with the potential for eye-popping involvement that many believe search ads lack. Publishers see this as a competitive advantage against search, which even with video tends to be limited by ads on the sides of results.

Despite this growing capability, and despite the awakening of interest among major corporations in display advertising, many Web publishers still feel trapped toward the bottom of the advertising system's internet food chain. To understand what this means and how it pushed audience labeling and targeting to new heights, we need to examine the forces that shape those who hold power. First, consider that two audience-rating companies, Nielsen and comScore, are central in an odd way to the development of the Web's power structure. You might wonder why the Web needs ratings firms if the sites themselves can audit the number of people visiting them. One reason is that early in the Web's history media buyers decided that they would not trust websites’ accounting of their individual (“unique”) visitors. Instead they accepted the television industry's model for learning about Web audiences: the survey panel. As the dominant players in that space, Nielsen and comScore place tracking software on computers in homes and the workplace. They use their findings along with what they know about those people (including general media use and shopping patterns) to make generalizations about visitors to particular websites. ComScore reports that its panel includes about two million people “under continuous measurement” globally—one million in the United States and one million distributed across more than 170 countries.9 Nielsen's panel size is smaller—several hundred thousand—and in fewer countries, but it emphasizes that it creates its sample by using the gold standard: choosing people based on a table of random telephone numbers.¹⁰

Neither Nielsen nor comScore will explain what kinds of workplaces let their computers be monitored by a ratings firm. (It's hard to imagine big firms and government agencies allowing it.) One doesn't have to dig into the firms’ survey methods, though, to realize that they fall short in auditing the traffic to websites. Comparisons between Nielsen and comScore data regarding the websites they rank as most popular typically do show much agreement on the relative rankings of sites in terms of visitors; it would be hard to miss that Google, Facebook, and Yahoo! should be up there. At the same time, the ratings firms often differ by tens of thousands—even more—on the specific audience figures they report for each site. It's not unusual to see, for example, even a site such as MySpace get very different Nielsen and comScore numbers. Nevertheless, media buyers use the ratings to give them a sense of the top-tier sites and their relative sizes.

But just as important as the large number of sites that show up on the Nielsen and comScore lists are the ones that are missing or are way down. While the number of sites that the panels visit is large, the fragmentation of the Web is such that even a survey of over a million people could not capture what writer Chris Anderson has called the “long tail” of the Web: those sites with relatively few visitors that accept ads and may actually be appropriate buys for certain types of advertisers.11 Advertisers are looking for what they call “scale”—the ability to purchase huge numbers of individuals who fit their targeting needs without the expensive chore of having to cherry-pick them across thousands of Web publishers. The online advertising world is therefore split between premium sites that advertisers recognize and a sea of others with audiences advertisers may want but with much less ability to access media buyers directly. In each category are sites that spend lots of money creating content (so-called legacy publishers) and sites that draw most of their material from visitors (so-called user-generated content) or from other sites. Many of the first type of site are traditional firms that make most of their money in print. Many of the second type of site are rooted to the Web itself. Each type of publisher can present scale to advertisers. The New York Times is a legacy publisher with around thirty million unique visitors per month to its nytimes.com website. Huffington Post, by contrast, started on the Web and reaches around the same number of unique individuals per month.

The fortunate publishers that score high on Web ratings charts do have an opportunity to parlay their numbers into advertising by selling ad space on their sites directly to advertisers. For ratings help, sites that might not be well represented in Nielsen or comScore rankings often turn to Quantcast, a measurement company that affiliates with over ten million sites partly as a result of its offer to audit and quantify their audiences. In return, Quantcast loads the computers of the publishers’ visitors with cookies that track what they do online. It uses the cookies to find “lookalikes,” which are people who seem to reflect the background, interests, and activities of an advertiser's primary targets. They are, Quantcast tells marketers, “audiences who look like your converting customers”—that is, the ones who buy or otherwise respond to the advertisers’ requests.12 It sells the ability to reach the lookalikes to advertisers and shares the revenues with the publishers who let it drop the cookies.

Web publishers that sell directly to advertisers use mixed business models. Some sites offer ads to advertisers on a cost-per-click (CPC) or cost-per-action (CPA) basis. That is, the sponsor pays only if a visitor clicks on the ad or (in cost per action) performs a subsequent deed, such as phoning the company or buying the product. Most publishers, though, sell advertising space on a cost-per-thousand-impressions basis. An industry group defines an impression as “a delivered basic advertising unit.”¹³ Every time an advertisement loads onto a user's screen, it will be counted as one impression. When publishers adopt the cost-per-thousand-impressions system they get paid whether or not the individuals click on the ads. They use this approach because even though the raw CPC and CPA prices are far higher than CPM costs, the percentage of visitors who actually click on ads is miniscule—far less than 1 percent. The percentage that carries out particular actions is far lower than that. The Google, Bing, and Yahoo! search engines, which charge on a CPC basis, do in fact yield a very low ad-click rate. They can make money charging advertisers on a per click basis because they send out billions of ads each day. Few publishers come close to matching that.

Publishers often try to increase the pages on their sites. They do it partly to increase their chances of showing up in search engine results and partly to give visitors additional reasons to stay. Keeping people on the site longer—the industry term for this is engaged—means the publisher will have more opportunity to learn about them and serve them ads. Publishers often take a host of steps to find out about their visitors, including incentives that induce these individuals to reveal their identity; even a thin amount of personal information such as a name, e-mail address, and ZIP code can enable the site to begin a personal file on an individual. For example, nytimes.com has sign-up data from the more than one million individuals who subscribe in one way or another to its service. It encourages nonsubscriber registration by making it a prerequisite to be able to comment on articles, receive news alerts, and sign up for free e-mail newsletters. The website requires registrants to enter an e-mail address and certain demographic information (ZIP code, age, sex, job industry, and job title). It also asks for household income. Meanwhile, nytimes.com tells advertisers that it “has an extensive database of reader-supplied information. Through reader surveys and registration data analysis, we can offer you access to your ideal audience.”¹⁴

Many popular sites, including nytimes.com, add to these data through voluntary surveys that they conduct of visitors on the websites, on the phone, or even through the mail. Nytimes.com has an arrangement with the LinkedIn social-media site to match nytimes.com registrants to their LinkedIn profiles and transfer that information to nytimes.com for its use in advertising.¹⁵ Some publishers, though evidently not nytimes.com, purchase data about their registrants from information vendors such as Experian and Acxiom and append them to their files. The data may directly involve the individual (profession, credit status, number of airline flights) as well as the individual's household (number of kids, age ranges, home value). Information purchased from other companies may help the site make lifestyle inferences about an individual. So, for example, Nielsen's PRIZM segmentation system might suggest that a person between forty-five and sixty-four years old living in the 19004 ZIP code area has a high chance of being a member of what PRIZM calls the “Upper Crust: wealthy, without kids.” People in that cluster “shop at Saks Fifth Avenue, belong to a country club, read Condé Nast Traveler, watch the Gold Channel, drive Mercedes SL Class.”¹⁶

Then there is the all-important behavioral data that websites create by placing cookies in visitors’ computers—even those who don't register—and making inferences about their interests and even their personalities based on the links they click and the topics they view. A person who tends to read articles about health might be tagged as having specific health interests; a sports reader might be placed in that bailiwick. Sites often hire analytics firms to perform exhaustive analysis of these sorts of data with an eye toward showing why advertisers should find their visitors particularly interesting and consequently pay a lot to reach them.

Nytimes.com reports that it hires Audience Science to track and analyze its site's browsing patterns. While the sentence is tucked away in its privacy policy (where the Times implies that Audience Science uses Web beacons and tags to carry out its work), a glance at what the company does explains why it is critical to nytimes.com's attempts to make its online audience attractive to advertisers. In Audience Science's own words, it analyzes “multiple indicators” to make inferences about visitors: “which pages and sections they have visited, what static and dynamic content they have read, what they say about themselves in registration data, which search terms they use, and what IP [internet protocol] data indicates about them, including geography.”¹⁷

All this can offer advertisers impressive detail when choosing to target people on the site. Moreover, audiences are involved in a continual process of data capture and message presentation. Advertisers and publishers gain usable information about site visitors at the same time they are selling to them. But even major sites such as nytimes.com, with a million individual visitors per month, have found that they can't sell their advertising positions by themselves. Nytimes.com may sell out its front page and other highly trafficked areas of the site. It may also sell “site sessions,” which allow an advertiser to “own all display banner positions,” except for the home page, that are seen by the targeted individuals.¹⁸ Nevertheless, many positions on the nytimes.com site, including reader comment areas, interest few advertisers and so cannot be sold at premium prices. An executive for the New York Times who spoke on the condition of anonymity estimated that these unsold positions constituted approximately 50 percent. He predicted optimistically that the percentage would actually decrease because advertisers would want to reach the select group of people who had chosen to pay for new digital versions of the newspaper. He and others agree that at many sites, even those of big-name off-line publishers, the percentage of advertising space they cannot sell directly is far higher, often close to 80 percent. Some in the industry call this unsold space “remnant ads,” though others take exception to the carpet-peddler image the term evokes and instead dub the ads “nonpremium.”

Because of this oversupply, virtually all sites link up with at least a few of the hundreds of advertising networks that have emerged to profit from the need for such intermediaries. As Chapter 2 noted, advertising networks came into being pretty close to the Web's big bang. They had developed in the 1990s as a revenue source for websites that didn't have huge visitor numbers and that were running out of venture-capital money. The networks’ strategy was to learn a lot about the behaviors and backgrounds of the individuals that their affiliate sites reached, to tag those people via cookies, and then to give marketers the ability to target particular types of people with clients’ specific ads when they returned to one of the hundreds, or thousands, of sites on the network. The networks would share their revenues with participating sites. So, for example, the Engage ad network in 1999 reported that it held over thirty-five million profiles. Some of the data came from users’ internet protocol (IP) addresses (part of which indicates geographical location). Other data were inferred about the people through their activities (what a person did on sites, as monitored by cookies and Web bugs). The activity-based approach to evaluating visitors and then sending ads to them became known as behavioral targeting. Advertisers could buy the right to reach them according to up to eight hundred characteristics. Executives from Engage argued that not only was the firm's segmentation approach useful for ad targeting, it even helped with the click-through rate. Its research found that advertising targeted through user profiling received substantially higher click-through rates than ads served only around content.

Engage eventually disappeared, but the number of networks grew. In 2008 Jupiter Research estimated that approximately 28 percent of online revenue was flowing through them.19 By then, the four advertising powerhouses of the internet age—Google, Yahoo!, Microsoft, and AOL—had networks that included their own properties and millions of other sites. Google in the early 2000s established AdSense, a network of websites that agreed to allow Google to serve “contextual” text ads on them based on Google's analysis of the content of the sites’ Web pages that the visitor was reading. By the late 2000s, it had morphed into the Google Display Network, which serves ads with rich media and video formats to millions of sites.

Apart from the Big Four, approximately three hundred other networks ply their services to websites. Web publishers in search of advertising revenues often affiliate with more than one. ValueClick and Adbrite are among the biggest. Many publishers allow more than one network to place ads on their sites in order to generate revenue from multiple sources. The Huffington Post, for example, allows Advertising.com (owned by parent company AOL), ValueClick, Feedburner, and Google as “advertising partners.” These and virtually all other networks use cookies to track the behavior of visitors within and across thousands of websites. In this way they can develop behavioral profiles of individuals and place them into segments that allow targeting of large numbers of people for ads—for example, one segment could consist of women between the age of twenty-one and thirty-four who live on the West Coast and who are in the market for a small car. Using cookies also enables ad networks to “retarget” individuals who had seen an ad on one site with an ad for that product on another site.

The Advertising.com network boasts that it is all about combining niche targeting abilities with the kind of scale unavailable on even the largest website. “Finding your ideal consumer is one thing. Finding millions of them is another. With our targeting solutions, you can extend your message across thousands of sites and reach your customers at scale.” In its pitch to advertisers, Advertising.com emphasizes the contextual and behavioral expertise it can offer from its huge number of visitors to AOL-affiliated sites. Advertisers can choose niche networks, such as its community, casual games, or retail channels. They can identify Web users based on behavior segments such as “style mavens” and “family chefs.” And they can “pinpoint your customers with other powerful targeting solutions.” For example, the company claims it can: “Develop a custom audience segment modeled after visitors to your site (Look-Alike Modeling); find households that have the greatest propensity to purchase specific products or brands (MRI Lifestyle Clusters); if you're sponsoring an AOL page, retarget consumers who have visited it (Sponsorship LeadBack); find your ideal female audiences on the sites they are most likely to visit (Subnet Targeting); find women who are searching for information about fashion or home & gardening; explicitly target households with females present (Age/Gender Targeting).”20

The Audience Science Network (which is separate from the work Audience Science conducts for individual websites such as nytimes.com) is equally enthusiastic about its ability to help advertisers. The company says it “accesses 200 billion data insights into 386 million people worldwide.” These insights about intent, it states, can fuel marketing that encourages “conversions”—that is, sales, intention to purchase, or other responses marketers want. In addition to behavioral segments, the firm offers targeting and retargeting based on demographics, location, time of day, internet speed, “and more.”²¹

Despite these promised benefits, marketers tend to have a love-hate relationship with advertising networks. One reason is that the many networks don't make the names of many sites known to their clients; publishers that sell their own advertising space often don't want advertisers to know that they also sell their ads more cheaply via networks. Moreover, with just a few exceptions ad networks do not guarantee the placement of ads on any particular sites.²² Consequently advertisers often don't exactly know where their ads appear unless visitors click on them. Sometimes ads appear in e-mails, on minor blog sites, or in other places on the fringes of the Web. Major advertisers tend to be particularly concerned that their ads not be placed among salacious content or content that might reflect poorly on their products. Under such a system, the challenge of delivering the number of promised impressions to the promised targets can be met simply by loading them up repeatedly on a small number of people. To prevent this, the best networks use a process known as frequency capping, which limits the number of times a computer gets served the same ad. But on other networks ads will sometimes be served over and over to the same individuals, much to their annoyance.

Still, marketers and their media-buying agencies like networks because they are extremely inexpensive on a cost-per-thousand-impressions basis—typically between fifty cents and a dollar.23 Many advertisers use this avenue as a way to simply spray the Web with ads in the hope of direct-response revenues. But major marketers with brand-oriented initiatives have also been using cheap network buys as inexpensive ways to reach out to huge numbers of people via display ads with the expectation that a certain percentage will click on the ads. Media buyers evaluate the return on investment through particular measurable activities that relate to their campaign goals—prospective customers reading parts of a minisite, asking for more information, providing e-mail addresses, and the like—that may encourage branding and move a customer toward a purchase. Increasingly, advertisers are erasing the traditional distinction between branding and direct response to assess instead various levels of engagement. General Motors’ Chevrolet division conducted a survey to determine the relationship between clicking on any of hundreds of features of its site and visitors’ interest in the brand and their intent to purchase a car. Chevrolet used these findings to create an index that assesses the return on investment it gets from people's click and mouse-over responses to different forms of Web advertising.²⁴

For Web publishers, the existence of advertising networks has been at best a mixed blessing. Sites that have no ability to generate revenues independently use networks to make money. When costs are low (for example, if the sites carry mostly volunteer blogs and pay little in salaries), the revenues from advertising networks might help them get by. But for sites with major expenses—often legacy publishers with off-line newspapers and magazines—advertising networks bring in money at shockingly low CPM levels, prohibiting publishers who might otherwise consider a switch to an online-only model from having sufficient resources to do so. A 2008 survey of four anonymous companies by the Bain consulting firm found that while the Web advertising they sold directly to advertisers yielded between $12 and $18 average CPMs, they were getting around $1 CPM for advertising sold through networks. More than 80 percent of the revenues of those firms came from the 20 percent of their advertising.

The anonymous New York Times executive mentioned earlier also said that to try to leverage as much revenue as possible from its advertising opportunities his company discounts ad positions that it cannot sell at premium prices. In paying lower prices, the advertisers are not guaranteed an exact date or placement for their ad. The executive also acknowledged that nytimes.com has an agreement with Google's Display Advertising Network (DAN) to auction off the website's unsold positions. (Google therefore places cookies into the browsers of nytimes.com visitors and tracks their behavior there and across the Web.) In distinguishing Google from other advertising networks he noted that although a bidder can specifically request the nytimes.com website, the DAN's AdSense platform allows his company to exert control over the kinds of ads served to the site. In general, he said, nytimes.com and a few other major Web publishers such as NBCUniversal Interactive are wary of advertising networks because they might serve ads for products that would not fit the publisher's image—for example, diet pills or products that claim to enhance breasts or muscular physique. The New York Times official portrayed AdSense as a way to gather advertising from small companies—local restaurants, theaters, and the like—that mesh with the site's audience but that the Times ad-sales staff do not have the time to contact themselves. The amount and percentage of revenues that nytimes.com receives from ads sold this way are confidential, but the Google auction prices obviously will not come close to the rates that the nytimes.com sales force gets for the site's premium and even discounted ads.

The declining amounts of money publishers make as they move from print to selling their own ads on the Web and to accepting ad-network prices are startling. A source familiar with newspaper off-line and online pricing (and whose example was confirmed by others in the business), and who agreed to be interviewed on the condition of anonymity, described the situation using 2010 figures: The average CPM rate for a major print newspaper is approximately $50 (major advertisers may get a substantial discount). The CPM rate for direct sales for ads on a newspaper's website ranges from $25 to $40 (ads with videos typically fetch the highest amounts). But any given company sells only about 20 percent of its advertising positions through direct sales. The rest is sold through ad networks at a CPM rate ranging from $2 to $4. The ad network gets 50 percent of the amount earned, so a paper's website ends up with CPMs of $1 to $2 for 80 percent of its advertising space.

To improve these dismal numbers, firms such as Audience Science encourage publishers to enhance the knowledge of their audience that they bring to the ad networks; Audience Science contends that better profiling can make CPM values fifteen times higher. Yet while publishers are plumbing more and more about their audiences—and hiring companies to help them—they are being thrust further and further down a food chain that diminishes their value and power.

A relatively new ad-selling environment, the advertising exchange, does not materially improve the situation much for publishers even as it excites marketers. Advertising exchanges are centralized markets for buying and selling audience impressions. Instead of going to a few of the more than three hundred ad networks and buying the right to reach a certain number of individuals with certain characteristics on anonymous sites, ad exchanges provide a digital marketplace whereby any party—networks, publishers, and even advertising agencies—can buy and sell the right to reach anonymous individuals. Some in the industry predict that ad networks will eventually disappear as all parties move to exchanges to sell their ad positions. In early 2010 there were about a dozen ad exchanges, and the biggest were owned, not surprisingly, by Google (Ad Exchange 2.0), Yahoo! (Right Media), and Microsoft (AdECN). Advertising exchanges are more efficient than ad networks because all the buyers and sellers can come together in the same places. Moreover, unlike most ad networks, ad exchanges often let media buyers know the site on which the impression will be served. But the real benefit of ad exchanges at their most high-tech in the eyes of their proponents is the ability to make a decision to reach certain types of individuals on the fly, wherever they are. A growing number of exchanges offer this “real time bidding” (RTB) on individuals, at virtually the moment they load the page of the site they are visiting. One industry observer called “capturing the individual consumer” through RTB in exchanges “the holy grail of targeting.”25

The rise of a market in impressions has naturally stimulated unprecedented data-collecting activity related to individuals. One investment firm's 2009 report noted that “contact information is now collected at virtually every step in a user's online experience (via registration pages, for example) and Web surfing behavior is tracked down to the millisecond—providing publishers and advertisers with the potential to create a reasonably complete profile of their audiences, and thus enabling the matching of a user with a user profile to enable robust, segmentation-based targeting.”²⁶ For example:

•  eXelate says it gathers online data from two hundred million unique individuals per month through deals with hundreds of websites that see an opportunity for revenue based on the firm's visitor analyses.²⁷ eXelate determines a consumer's age, sex, ethnicity, marital status, and profession by scouring website registration data. It also tracks consumer activities online to note, for example, which consumers are in the market to buy a car or which are fitness buffs, based on their internet searches and the sites they frequent. It gathers the information anonymously using tracking cookies placed on the hard drive of a consumer's computer when that individual visits a participating site. Through what eXelate calls its Targeting eXchange, advertisers bid on the right to target the consumers with those cookies (as well as those of other firms), and eXelate shares this revenue with the sites that provided access to the data.28

•  BlueKai, an eXelate competitor, also strikes deals with thousands of websites to use cookies to collect anonymous data about visitors’ activities. From that information, it makes inferences about the kinds of purchases that might interest those people—their “intent to purchase.” The company also operates what it calls the BlueKai Exchange—“the world's largest data marketplace”—where it auctions its own cookies and those of other firms. BlueKai claims that it can target more than two hundred million users based on “actionable audience data” at “any stage of the purchase funnel.”²⁹

•  The data firm Lotame tracks comments that visitors leave on websites and from these creates individual profiles based on what site visitors have written about movies, say, or about parenting and pregnancy. Lotame packages the data without determining individual names but attaches to them unique cookies, and then goes to exchanges to sell the profiles and access to these potential customers.³⁰

Marketers often have cookies of their own that they've created for people who visit their product sites. Consider Procter and Gamble's Pampers diaper line, which has a “Pampers Village” website aimed at mothers. P&G scarfs up loads of information about visitors, from data requested at registration to be part of the site's “community” to behavioral data regarding what they look at or even write on the site, as well as to information from outside data providers about the individuals whose names it recognizes from registration.³¹ To reach these people elsewhere on the Web, P&G might bid on an ad exchange to buy individuals who match its cookies. Cookie matching allows the cookie seller (for example, BlueKai) to detect in milliseconds whether the computer that has its cookie also has a cookie from Pampers. If there is a match, the company buying the impression not only obtains the information that the seller (such as BlueKai) has about the individual, it now can link that knowledge to information that it owns about the person. And there's more: data firms have found ways to link off-line data to e-mail addresses, registration information, and similar personal facts that people reveal about themselves online. For example, eXelate in 2009 began to offer advertisers the ability to append to eXelate cookies data from Nielsen that includes information from the Census Bureau, Nielsen's own research, and research by other consumer-research firms, such as Mediamark and Experian-Simmons.32 P&G's Pampers is also involved in these online/off-line linkages. The firm's privacy policy says that it links information it gathers about visitors online to data “we collect about you from other sources, such as commercially available sources.”³³

So who are the winners in this digital selling-and-buying food chain? Clearly, the data-exchange firms are big winners. So are the major ad networks, even in the exchange world, because they have enormous amounts of data via cookies that they can sell to advertisers, sometimes merging their cookies with those of the data exchanges. Google presents an interesting case of the owner of data-exchange and ad networks that changed aggressively to meet the competition. In the early 2000s news surfaced that Google was gathering up huge amounts of information about its visitors’ search activities and sometimes linking it to profiles gleaned through registration on its Gmail and other programs. Yet Google implied it was using all the data for a kind of actuarial research—to hone its ability to link search terms and website writings with search terms and ads that yielded lucrative clicks. Nevertheless, the revelations of the data capture resulted in calls by privacy advocates for short time limits for this type of storage. Google did, however, manage to avoid the privacy controversies that were roiling around its e-mail and Web-publishing counterparts, because for most of the decade the company insisted that it did not use personal or historical information about an individual either to influence that person's search results or in its AdSense network, where it served ads based on the context of a Web page.

The company changed course as part of its goal to become highly successful in the display market; these efforts accelerated after it purchased DoubleClick, an ad-serving company that also ran an advertising network.³⁴ It seemed clear that Google was adapting quickly to compete with, and to try and outstrip, its rivals with audience information. Although it kept its promise of not sharing users’ personally identifiable information with its advertisers, Google increasingly ramped up its use of behavioral, demographic, and geographical data in serving its ads on the Google Display Network. The network also opened its tags (and therefore its cookies) to other online target marketers it “certified”—Turn, X+1, Data Xu, Rocket Fuel, and Appnexus, for example—as a way to get extra revenue. Significantly, Rocket Fuel presents itself as delivering audiences in ways that Google itself has never publicized, but for which Google's network and cookies are facilitators. “We build designer audiences for your campaign by buying individual impressions of the users,” says a Rocket Fuel promotional document. “The Rocket Fuel platform is integrated with many of the leading third-party data providers—allowing us to leverage demographic, interest, lifestyle, past purchase, behavioral, in-market, social, purchase intent and search data… . We go beyond other audience targeting technologies by layering multiple unique data sources.”35

Where do media-buying agencies stand in this hierarchy? In the face of the hundreds of millions of dollars made on display advertising by networks and exchanges owned by Google, Yahoo!, and Microsoft, declaring media-buying agencies winners should not imply they were in the same revenue league. Yet in view of the precipice on which large media-buying agencies and their owners—WPP, Publicis, Omnicom, and Interpublic—found themselves in the early 2000s, they remarkably have managed to find ways to remain relevant to clients. For a number of years, the Web's structure had threatened to make media-buying firms obsolete in the digital space. Their clients the advertisers could easily purchase blocks of search terms and banners directly and automatically from the networks selling them. WPP chairman Martin Sorrell famously called Google a “frenemy” amid worries that it had both the technology and the data to serve as the kind of guide to reaching audiences that had been the purview of WPP's media buyers. As a defensive measure, WPP and its counterparts began to invest in tools that would help their clients in ad targeting, buying, optimization, and measurement. A critical competitive advantage they enjoy is deep knowledge of their clients’ target audiences. That includes information about what individual customers buy as well as access to the cookies of customers or potential customers who have visited client sites such as Pampers Village. While clients share such knowledge with their media-buying agencies, they would not want to reveal it to ad networks such as Google or to data providers such as BlueKai, because the networks and data firms might use the information in ways that could compete with the advertisers confiding in them.

Media-buying agencies’ desire to help their clients use their cookies most effectively meshes with the agencies’ interest in developing businesses that allow them to eke increasing revenues from their client relationships. The two goals have led media-buying agencies to join or create buyer-driven real-time ad exchanges aimed at efficiently auctioning cookies. Also called demand-side platforms (DSPs), they are controlled by the agencies even though on many platforms any publisher or network can submit a bid for a certain number of the cookies being offered. In a seller-driven—“supply-side”—ad exchange (for example, Google's DoubleClick Exchange) the owners (in this case Google) have the best views of the advertising space that exists in the online world and of the auctions that are taking place throughout. That means Google knows the difference between what the sellers of the ad space want to charge and what buyers will bid. Google has the advantage if it wants to get into the market, while media buyers have the disadvantage, as they see only the asking price from the publishers, the networks, and the exchanges. By contrast, with DSPs the agencies’ media buyers know both numbers; the exchange is fully transparent to them. DSPs also give the media-buying firms control over unique identifiers in cookies that publishers and marketers set through the bidding process; they don't have that kind of power in seller-owned exchanges. They can consequently plant their own cookies in the computers of the people whom they have bought from publishers. All this makes the DSPs quite valuable to their marketing clients.

Web publishers, by contrast, continue to get pushed around, and down, the power ladder. Leaders representing the networks, data exchanges, advertising exchanges, and media-buying firms continually exhort the publishers to keep the system humming by giving up more and more information about their visitors. The chief operating officer at the digital agency Neo@Ogilvy warned publishers that increasingly specific data was the only way to stop the downward spiral of Web ad prices. “The saviour of this commoditization [of advertising space] is data,” he contended.36 Other executives contend that publishers should marry such detailed information with real-time bidding (RTB) to get the most revenues from their visitors. Pubmatic, a firm that aims to help, puts the situation this way:

Currently, most digital media buying is done based on assumptions about certain audiences. For example, audiences bought through ad networks and ad exchanges are often purchased in buckets [that is, simply in large numbers of uncategorized people] or by segment. How the audiences are categorized in certain segments depends on who is selling them. And while some audience sellers do a better job of segmenting users than others, so long as individual impressions are being grouped into a bucket and sold at a pre-negotiated price, they are not being fairly valued and are often sold at under-valued prices.

In Example 1, a luxury car advertiser is looking for a very specific audience type and is willing to pay a premium price to reach a specific user that is highly qualified. The more qualified the user, the more the advertiser is willing to pay. On the right side of the example below, the advertiser (or rather the technology company placing bids on behalf of the advertiser) can see the unique characteristics about the user and therefore is willing to pay a $3.90 CPM to target that user. [A graphic notes that the user has visited multiple auto sites within the past few days, is currently—at 9:30 am—reading a vehicle financing page, and lives in Aspen, Colorado.] On the left side, the same user would have been bucketed into an auto-buying segment and priced according to the segment price, which [at $1.75 CPM] is far lower than what was paid via RTB for the individual.37

The reason for the higher price, Pubmatic explains, is that more information about users leads to improvement on click-through rates and conversion rates. Turn, a network that trades in real-time impressions, claims that its advertisers are seeing up to 135 percent improvement on the former and 150 percent improvement on the latter.³⁸

All this sounds great, except that publishers face several dangers that are built into the process. The first is the CPM price. David Cohen, the head of digital resources at the Universal McCann subsidiary of Interpublic, noted that the average price a publisher receives on Google's DoubleClick exchange is under $1. He said he didn't know the amount that nytimes.com receives when it sells advertising space on the exchange, but he doubted that it was much different.³⁹ Yield-optimization firms such as Pubmatic and X+1 try to help publishers by serving as automatic bid gatekeepers that evaluate all bids for publishers’ impressions (including non-real-time bids from networks and advertisers) in real time and select only the highest bids. Nevertheless, the increase in CPM that Pubmatic illustrates still doesn't come close to the more than $10 CPM that publishers make selling their premium ad positions by themselves. Publishers’ yield is affected still further by the cut that Pubmatic takes in return for its services.

A second difficulty for website publishers is the danger of data theft. Some advertisers and networks have been known to try to place their own cookie tags or pixels on the publisher impressions they buy so that they can place cookies in the machines of the individuals who load the ads and then retarget them without having to pay the publisher.⁴⁰ “Your data is being stolen,” eXelate warns in a PowerPoint slide show aimed at publishers.⁴¹ Tom Hespos, president of digital-marketing agency Underscore Marketing, has urged publishers to consider that demand-side platforms are pretty well set up to allow advertisers to drain value from Web publishers. He points out that “agency-side audience networks are leveraging the targeting data in which publishers have invested to target and segment audiences across the Web.”42 Interpublic's DSP, named Caedreon, has developed an ethics code that explicitly says it will not adopt such tactics to steal publishers’ audiences.⁴³ Other DSPs have not announced this prohibition. Pubmatic, eXelate, and other firms claim they have technologies to protect publishers from audience theft if the publishers work through them instead of going into exchanges on their own. But the need to lean on these third parties further erodes the independence of publishers in the digital environment.

Perhaps the most significant danger for publishers is that selling individual impressions has resulted in a marketing perspective that the publishing sites on which they appear are unimportant; the goal is to reach people with particular characteristics wherever they show up. Agencies’ decoupling of audiences from context is a profound change. Traditionally, advertising executives often tried to place ads where the “environment” of editorial matter or programming would fit with the tenor of the commercial message or the personality and status of the product.⁴⁴ So, for example, food ads in magazines would best be placed among recipes, while creators of television commercials generally want them to air on the types of shows that they believe draw attention and sales.⁴⁵ In the digital environment, this concern has narrowed to not wanting to have ads served near salacious content that might hurt the brand in the eyes of the audience. Apart from that concern, as an eXelate executive commented to ClickZ News, “who a user is is becoming more important than where [users] are.”⁴⁶

Publishing executives are beginning to recognize the exchange-driven devaluation of the website context as a danger to their websites’ long-term health. Wenda Harris Millard made just that point in a keynote address at the Interactive Advertising Bureau's annual meeting in late February 2008. Millard, the incoming Bureau chair, was at the time the president of the media division at Martha Stewart Living Omnimedia. She was also a former high-level Yahoo! executive. Millard excoriated the new digital marketplace for harming online publishers. She noted that Madison Avenue was “repaving itself” as agency giants resurged to dominate many areas of digital advertising, including planning and buying. She added that DoubleClick, Microsoft, Yahoo!, and other firms were fundamentally altering the media-buying landscape via their automated exchanges. Their bartering process will make sites that carry advertising interchangeable, she said. The result: a race to the lowest price without any concern for a site's personality, its history, or its credibility with visitors. Comparing the new marketplaces to commodities traders, she cautioned that publishers “must not trade our assets like pork bellies.”47

The warning prompted a response at the meeting from Michael Rubinstein, who at the time led DoubleClick's ad exchange. He disputed Millard's contention that automated buying and selling would necessarily decrease the cost of advertising. The key to correctly understanding the situation, he said, is to change the metaphor. Forget pork bellies; think gems. Gems, he noted, are also traded in markets, yet each one has distinctive characteristics and therefore a particular value. “We [at DoubleClick] like to think of our publisher impressions as diamonds,” he said, “not pork bellies.”⁴⁸

The exchange encapsulated anxieties swirling at the center of the advertising industry. Randall Rothenberg, executive director of the Interactive Advertising Bureau, tried to make peace between the publisher and the marketer, a role his organization (previously the Internet Advertising Bureau) had played since the time of Procter and Gamble's historic 1998 conference. He framed the tensions as long-standing; he said they went back to the recovery from the dot-com collapse in 2002. They involved, he noted, basic fights over audience measurement and the best ways to prove audience attention at a time of the enormous changes Millard had mentioned. He exhorted all the actors involved that they had “better stop marveling at all the creative destruction going on around us.” Instead, he said, “It's time to write the constitution that will help us live in peaceful and profitable existence.”⁴⁹

Rothenberg undoubtedly knew this was easier said than done. He might also have understood that the gems metaphor didn't really change Millard's fundamental argument. Where gems appear is increasingly not as important to advertisers as getting the gems. Moreover, as the new advertising norm progresses, the pork-bellies-vs.-diamonds metaphor raises questions about the value of individuals. Are individuals like pork bellies, with their value and the value of sites delivering them pushed down incessantly? Or are there ways to make at least some of the people into diamonds, with a resulting rise in the price to reach them? As we will see in the next two chapters, the precarious position in which Web publishers have found themselves as a result of the new media-buying logic pushes them in three socially troubling directions: it leads them to pursue the kind of personalization toward useful targets or waste that database-driven media buyers demand; it causes publishers to create privacy policies to hide particulars of buyers’ audience-tracking and targeting activities from visitors to their sites; and it encourages them to further retreat from longtime professional norms in the interest of packaging personalized advertising with personalized soft news or entertainment.