I have never yet been afraid of men who set up marketplaces in the middle of their city, where they lie and cheat one another.
—Herodotus, Histories
JUNE 15, 2012
120 milliseconds is about one-third of the time it takes a human to blink an eye.
That’s what we had to play with. That’s how much time an FBX partner had to return a bid and an ad when requested to by the exchange, an auction process that would take place fifty billion times a day, or about half a million times per second. Any longer than that, and it threatened to delay the Facebook page load. If you should ever be standing in a postlaunch Zuck demo, and the FBX ads load a tick slower than the rest of the page—or, worse, make the entire page load slowly—then you are on the fast train to a job at Dropbox, and don’t let the Facebook door hit your ass on the way out.
Those 120 milliseconds included the network latency, meaning the time elapsed when the bits flowed out of the Facebook pipe to the outside Internet, and to wherever the FBX partner’s machine was located, and back again.
This presented a serious and seemingly insurmountable technical challenge. Because Facebook was then being served for the entire world out of data centers in North Carolina, California, and Oregon, European marketers and their mountains of data had to talk to machines in the United States, often on both coasts. In order to comprehend the feasibility, I calculated the spherical distance between North Carolina and Amsterdam (where much of European tech is hosted on server farms), to see if it was even theoretically possible to respond to ad auction requests. Light travels fast (299,792,458 meters per second), but not fast enough. It was 23 milliseconds one way, and that was assuming lossless fiber-optic cable running directly on a great circle route from Facebook’s machines to the advertiser, a ridiculous simplification of the reality. Pinging the distance from an East Coast machine revealed a more realistic 60-millisecond one-way travel time. There was no way to beat the relativistic bound, and for as long as I was at the company, we lost money in Europe as many of those requests simply timed out, and FBX did not participate in the auction.
If you’ve read Michael Lewis’s Flash Boys, which details the ultrafast, high-frequency trading that takes place on the stock exchange, you’ll recall his prefatory riff on a hedge fund that ran a fiber-optic cable from New York to Chicago. The goal was to shave a few milliseconds off the regular Internet route between the two financial hubs, and gain a split-second advantage over other high-frequency shops. The cost was reputedly $300 million, which indicated the stakes involved.
You’ll see that my analogy to Wall Street for programmatic media wasn’t merely historical; it was an exact comparison reflecting both business and technical realities. FBX was the Flash Boys of media, trading quanta of human attention at the speed of light. Every time you load a new page in Facebook, not to mention most of the Internet, optical signals crisscross the globe to hundreds of waiting machines, all announcing, like some phalanx of royal trumpeters, your impending arrival.
So what happens to those globe-spanning bid requests, assuming they don’t drown in their trans-Atlantic undersea cable? The people listening for your presence in billions of such requests per day are known as “demand-side platforms” (DSPs), and they’re the stockbrokers of this real-time media world, working in the employ of the advertiser or agency who wants to sell you something.* The DSP quickly unpacks the bid request, and queries its data for anything it knows about you: sites you’ve browsed, shopping carts you’ve abandoned, that airfare quote you got and never acted upon. They’re all there and returned within milliseconds via state-of-the-art databases hyperoptimized for the purpose. Much of that data isn’t even directly observed by the advertisers in question. Companies have done deals for the right to rent a little piece of the webpage on sites of commercial interest, just enough to touch your browser and see who you are. These data brokers put you in some targeting segment—for example, “travel intenders” (i.e., people about to spend money on hotels)—and then resell you as “third-party data.” Since everyone has a pseudonym for you via your Web browser, and that browser is known to Facebook, Google, and everyone else, that data can be used to target you. This is all anonymous (in the sense of Facebook data never leaking out), but it’s everything you do online. All of it is more spice in the targeting sausage going through the advertising meat grinder, and is trafficked millions of times per second (whether you know or like it or not).
The real-time, millisecond bids from Facebook Exchange then fed into the regular ads auction that Facebook was holding on behalf of non-FBX advertisers. At which point Facebook, with all of its vaunted data, became merely one more DSP in the auction, just one more broker doing its business. The difference was, the rest of the Facebook Ads system had Facebook’s set of data, while the FBX advertisers, however many there were in that auction, had their own data from the non-Facebook world. Since both the Facebook Ads system and the outside world now received a request for an ad via FBX, we had put outside advertisers on an equal footing with Facebook itself, and may the best man (or bid) win.
I saw it as a feature. Facebook saw it as a bug.
This was FBX’s unforgivable sin: it allowed outsiders the same ability to optimize and target ads alongside the Facebook Ads machinery, choosing whom to target, on which page, and at whatever frequency, just as Facebook did. That equality of outside parties and Facebook was something Ads management could not abide. For all the bluster about Facebook’s wonderful data, management was unwilling to go head-to-head with outside data, because the managers suspected they’d lose. Which they mostly did: the bids of FBX advertisers were typically way higher than bids brought in by the rest of the Ads system, meaning FBX advertisers tended to win the auction for users’ attention more often. Of course their bids were higher: FBX bidders knew you had just spec’ed out a Jack Spade handbag or had bought diapers last week; Facebook by contrast knew you had liked Jim Carrey’s fan page a year ago. Whose bid for an ad was going to come in at $30 CPM?*
Note: There was nothing magical about FBX. It was merely sophisticated plumbing. The point was that joining Facebook Ads, via a real-time exchange, to a world of data it didn’t previously have access to was the fastest and most sophisticated way to get that data into Facebook. Custom Audiences could also bring the exact same data into Facebook, but nobody who dealt in sophisticated targeting for a living wanted to use Facebook’s clunky infrastructure to facilitate it. To cite loose technical analogies, CA versus FBX was like faxes versus email. Surely Silicon Valley’s most aggressive company would see the historical inevitability of programmatic exchanges and their technical superiority.
MOVE FAST AND BREAK THINGS! they told us.
FORTUNE FAVORS THE BOLD! the poster said.
But here is where I shamefully displayed my naïveté. For someone who had spotted the resemblance between Goldman Sachs and Facebook within an hour of meeting the latter, I had forgotten my lessons learned at the former. When it came to monetization, Facebook had no interest in real innovation. It liked its faxes. Like any large company, Facebook would always aim to create monopoly pricing power and maintain information asymmetry, rather than drive true innovation. If Facebook played with the outside world, it always played with loaded dice.
Recall the depths of the credit crash at Goldman back in 2008. Goldman could have pushed for the obvious technical step of trading credit derivatives on exchanges, which would have resulted in greater volume and greater transparency, and taken the regulatory heat off. But would Goldman cede its information asymmetry in the form of the trading flows that it, and only it, saw? Would it cede the ability to more or less arbitrarily set prices for credit risk, alongside a tightly knit network of brokers, effectively manipulating the market to its own benefit, rather than offering an open one?
Of course not. And neither would Facebook, when it came down to it.
I wasn’t thinking about that on June 15, though. Because that was the day the first successful bid on FBX finally was made, with one of FBX’s better partner companies, TellApart.* We had pulled it off—we’d built an ads exchange and a parallel ads system to Facebook in about five weeks of engineering work, and maybe two months total of product ideation. The mood was high at the FBX table. Trackie the Owl would finally be able to eat his long-sought mouse. Initial volume was of course light, and what would preoccupy us all for the next six months was increasing the volume of bids and money as quickly as possible and making FBX the centerpiece of an entire vision of how Facebook should monetize. Eventually, though, the race became simply to make FBX too big for even Facebook’s murderous management to kill. FBX was fighting for its life almost from the moment it was born.