For a glimpse of the future, consider what happened the lazy afternoon of May 6, 2010. By that time, the percentage of securities trades initiated by HFT programs had ballooned to an astonishing 60 percent.1 For all practical purposes, machines, not people, populated markets. Your innocent E*Trade order for one hundred shares of Google was a mere snowflake in this perpetual blizzard, executed mainly as a courtesy to perpetuate the illusion that you can participate in the American dream.
Starting at precisely 2:42 p.m., the Dow Jones Industrial Average plunged in a matter of minutes, off more than a thousand points, or 9 percent, from its opening that day. Over $1 trillion in asset value had disappeared by 2:47. That’s real money—your and my savings, retirement accounts, and school endowments. The stunned traders on exchange floors around the world could hardly believe their eyes. It was as if God himself had taken a hammer to the market. Surely this was some sort of horrible mistake?
It wasn’t. It was the result of legitimate HFT programs doing exactly what they were designed to do.
It took the SEC (U.S. Securities and Exchange Commission) nearly six months to sort through the electronic wreckage and figure out what had transpired. While the commission’s conclusions are somewhat controversial, which itself is an interesting commentary on what happened, the problem started when a money manager at a large mutual fund company (reportedly Waddell & Reed of Overland, Kansas) placed an order to sell a sizable quantity of stock in a highly diversified form known as the S&P 500 E-Mini.2 Ironically, Waddell & Reed is the antithesis of a quick-buck artist. Quite the contrary, it’s known for an investing style called “fundamental analysis,” buying and selling stocks slowly and methodically based on the performance of the underlying companies.
The hapless money manager wasn’t trying to do anything unconventional. He simply placed a substantial but otherwise routine order to sell seventy-five thousand contracts as soon as practical, at a rate not to exceed 9 percent of the trading volume over the past minute, in an attempt to ensure smooth execution of the order. Then he turned his attention to other matters.
The problem was that there weren’t enough buyers in the market for that particular security at that particular moment and, with nobody watching, prices dropped precipitously. As momentum built, and other programs automatically executed “stop-loss” orders to sell at any price, the denominator of that percentage grew and grew.
But that’s only the start of the story. Safety alarms, responsibly incorporated into HFT programs all over the world, went off. Detecting unusual market fluctuations, some began dutifully unwinding positions at a furious pace to protect their patron’s money. It was a full-on instant electronic bank run. The more aggressive ones, sensing a rare opportunity, smelled blood in the water. Interpreting the frantic buying and selling of their electronic counterparts as prey on the run, they traded furiously on their proprietary algorithms’ predictions that the generous spreads would quickly evaporate. Due to the unprecedented volume of transactions, reporting systems fell behind, injecting false information into this pileup. Apple’s stock price inexplicably soared to over $100,000 per share, while Accenture crashed to the bargain basement everything-must-go price of 1¢ per share. Really. Meanwhile, back in the real world, the sun was still shining and both companies were peaceably going about their business as usual.
In a moment as dramatic as a Hollywood cliffhanger, a single unassuming party saved the day with a simple action. The Chicago Mercantile Exchange, an out-of-town sideshow to the dominant market makers in New York, simply stopped all trading for a fleeting five seconds. That’s right, a little longer than it will take you to read this sentence. A flash to you and me, but an eternity for the rampaging programs brawling as ferociously as they could. That was sufficient time for the markets to take a breath and for the HFT programs to reset. As soon as the mayhem ended, the usual market forces returned and prices quickly recovered to near where they had started just a few short minutes ago. The life-threatening tornado evaporated just as suddenly and inexplicably as it had appeared.
While the story may seem to have a happy ending, it does not. Confidence in the institutions we trust to shepherd our hard-earned savings is the bedrock of our financial system. No blue-ribbon presidential panel or SEC press release can restore this loss of faith. It can and will happen again, and that threat hangs over our every spending and savings decision. Investors can no longer go to sleep secure in the knowledge that when they awake, their nest egg will still be intact and incubating. The sorry truth is that its fate is in the hands of the machines.
These electronic wars aren’t confined to the financial sector. They are becoming a standard part of our commercial landscape in a wide variety of areas. But you don’t have to worry about them spilling over into your home. They already have, though in a more benign way.
On an unseasonably cool winter afternoon in Silicon Valley, I visited a friend who works at a hot new company called Rocket Fuel. Flush with a fresh infusion of $300 million from a secondary offering, Mark Torrance, chief technology officer, took a break to meet with me and discuss his company’s business. His own customers have virtually no idea how he does what he does, but they certainly like the results. No, the company does not make fuel for rockets—it buys space on websites and display ads for household names like Toshiba, Buick, and Lord & Taylor. Sounds simple enough, until you consider how it’s done. The company describes itself as a “Big Data and artificial intelligence company focused on digital marketing.”3
You might wonder who decides which ads you see when you load a web page on your computer. You may assume that the owner of the website sells the space to the advertiser, possibly through an intermediary, like Rocket Fuel. But the truth is far more complex.
When you load a web page that contains ads, a monumental battle ensues behind the scenes in a snap of the fingers between a menagerie of exotic synthetic intellects. In the second or so between the time you click on a link and the page actually appears on your screen, hundreds of transactions ricochet around the Internet furiously gathering an astounding array of details about your recent behavior, estimating the likelihood that you can be influenced by one of the available advertisers, and engaging in a flash electronic auction for the right to make an impression on you. (Each display of a single ad is, in fact, called an “impression.”) Rocket Fuel is one of the most heavily armed warriors in this electronic skirmish.
Let’s start with the groundwork. Just about every time you visit a website, click on a link, or type in a URL, the page that you load notifies one or more parties other than the site you are visiting of your arrival. How this is done isn’t terribly important, but it does illustrate how the historical academic roots of the Internet have been repurposed for commercial purposes.
You may be aware that a web page actually contains links not only to other pages but also to files that display the pictures you see within the boundaries of the page, or “frame.” When pages are slow to load, you may notice these separate links flash by briefly, usually in a status line at the bottom of your browser window. They may come from the same website that you are visiting, but they often come from elsewhere on the Internet. Each picture has specific dimensions, usually measured in pixels, short for “picture element.” A pixel is basically one dot in a picture, with a color and brightness. So the more pixels in an image, the larger and/or more detailed the picture is. (You may have encountered this term in the form of “megapixels,” or millions of pixels, which is used to tout the quality of digital camera images.)
Early in the development of the Internet, someone made the clever observation that a picture on a web page could contain a single pixel, making it essentially invisible to you. Why display this if you can’t see it? That’s the whole point. You can’t see it, but that single pixel may come from anywhere, in particular from someone who would like to take note of when and from where you visited that particular page. Because the pixel comes from someone else’s server, they get the automatic right to make a notation, usually on your hard drive. These notations are tiny files that go by the colorful moniker of “cookies.” You can prevent this from happening with a setting in your browser, of course. But almost nobody does because it makes it hard to use lots of common features of websites. Also, the obscure web browser option to “block third-party cookies” means nothing to most people. Sounds like someone is withholding a tasty snack.
So what’s in these cookies? Usually nothing but a unique identifier in the form of a big number. The important information is kept on the server of the party depositing the cookie. They would never entrust such valuable information to you because you might inadvertently share it with one of their competitors. You can think of this identifier as the electronic equivalent of gently sticking a Post-it note on your back so they can tell if they ever see you again.
And see you again they do, as you move around the Internet, browsing websites, clicking on links, reading articles, and buying products, because these parties have put pixels all over the place. As a result, they can construct a remarkably comprehensive profile of your habits—what you like and don’t like, where you live, what you buy and from whom, whether you travel, what ailments you may suffer from, what you read, watch, and eat. But as extensive as this portrait is, it omits one crucial detail: who you actually are. They can construct a detailed picture of a single individual without knowing that person’s name, face, or other identifying details except that you are using a particular computer.
Now, you might wonder why the website you are visiting is letting all its friends put sticky notes on your back. For the simple reason that it benefits from it. Sometimes the website is paid in information: the parties collecting the data can give it lots of useful statistics about the demographics and personal characteristics of its visitors. But more often the website you are visiting wants to be able to advertise to you in the future after you leave the site. And access to the third party’s vast trove of tracking data allows it to do just that (for a price, of course).
You might wonder who the parties are that trail you around the Internet. Some are household names like Google and Yahoo; others are newcomers like Rocket Fuel, which Mark Torrance estimates has placed cookies on about 90 percent of all personal computers in the United States. To see why these cookies are so valuable, you need to understand the power of cross-referenced information. Simple facts, useless by themselves, can become very precious when combined. Together, these facts can be used to assign you to what’s called an “affinity group” that indicates your preferences or likelihood to buy a particular product.
For instance, if you read vegetarian recipes online, you may be much more likely than the average person to be interested in trying out a new yoga studio in your neighborhood. The chance that someone random will click on an ad for a golf vacation may be one in ten thousand, but if you are male, it may increase to one in a thousand, and if you’ve looked up who won the Masters Tournament, it may rise to one in a hundred. If you watch the entire Twilight movie trilogy, you may be inclined to purchase its soundtrack, but if you also watched Cosmopolis and Bel Ami, you may be likely to buy a magazine featuring an interview with Robert Pattinson (who starred in all these movies).
Or, most important, if you recently visited a web page for a specific product but didn’t buy it, say, a certain model of running shoes, you are far more likely to respond to an ad for that product if it is shown to you in the near future. The problem is that once you have left their website, the makers of those running shoes have no way to communicate with you again. So that’s where the parties depositing the cookies on your computer come in. When you turn up somewhere else, for instance, on a website to make a dinner reservation, they recognize you as the same person who looked at the shoes last week, and can show you an ad reminding you of your interest. This, called “retargeting,” is one of the most valuable forms of online advertising today.
Companies like Rocket Fuel have built elaborate mathematical models to predict how likely you are to respond to a particular ad from one of the advertisers that they represent. And they know, statistically speaking, just how much you are worth to each of those advertisers. So they know how much the advertiser can afford to pay to put a specific ad in front of you when you load a page.
And that’s where synthetic intellects come in. Keeping this analysis up to date is a task of monumental complexity far exceeding human capabilities. To do this well, they must collect and analyze a staggeringly large amount of data on a continuing basis. But machine learning systems with access to enormous computing power and data storage are equal to the task. They are perpetually sifting through this river of information, panning it for gold in the form of valuable correlations and waiting to jump into the fray when you next visit a page, regardless of where that page happens to be.
The problem is that the synthetic intellects of all the other parties that have cookies on your computer are doing the same thing. Each of them represents a differing collection of advertisers, and each predicts different values for showing you different ads on different parts of different pages with different browsers at different times of day.
Now let’s turn this around and look at it from the perspective of someone who simply wants to operate his or her own website and make money by selling advertising. Selling ad space to each individual advertiser is entirely impractical, except for a handful of the largest and most successful websites. Even selling the space to intermediaries (like Rocket Fuel) that represent multiple advertisers would be a nightmare. So elaborate electronic ad exchanges have sprung up to perform an actual price auction for each ad that quietly appears on the pages you load. The operators of websites simply consign their available inventory of ad space to the ad exchange. The intermediaries also sign up, and the games begin.
When you load a page, and it calls for a specific size of ad, this information is transmitted to the ad exchange. It is immediately put out for bid to the intermediaries, who look to see if they have a cookie on your computer. If so, they perform an elaborate evaluation to estimate how much they want to pay for that opportunity, taking into account every time they have ever seen you, where you have been, and what you have done in the past. They also consider which website you are currently visiting, what content is on the page you are viewing, and how likely you are to do business with the advertisers available in their portfolio.
Now things get complicated. The intermediaries might also purchase information about you from other companies that aren’t in the advertising placement business but have agreed to share cookies with them for a fee. Even at the lightning speed of the Internet, it isn’t practical to perform multiple auction rounds. So each bidder selects a particular ad from its roster and makes a single best offer. It also tells the ad exchange which advertiser’s message it plans to display, because the websites serving the ads don’t want just anything to appear on their pages. (For instance, a site catering to children may refuse ads for certain products targeted to adults, such as promoting casinos, even though the person viewing the page may be a good prospect for such. Or a diabetes information site may not want to show ads for sweets.) But nearly all sites refuse to show ads from their competitors. Finally, the ad exchange awards the opportunity to the highest bidder but charges it only the price of the second-highest bidder (which encourages participants to place their best, highest bid).
So, after quite probably expending more computing power than was used to put the first man on the moon, an ad seamlessly appears on the page you are loading … offering special vitamins for your cat to help fight feline leukemia. Amazing—how did they know that you just got a cat? Yikes, if they figured that out, what if they’re right about the leukemia too?
During a recent conversation, Rocket Fuel’s CEO, George John, pointed out to me the irony that the art of persuasion, something you might reasonably regard as a uniquely human endeavor, is better done by synthetic intellects. Numerous customer testimonials on the Rocket Fuel website remark how much better Rocket Fuel is at spending their ad budget than they could possibly be themselves.
You may notice that I’ve glossed over one important question: how do the bidders know what showing a particular ad is really worth to the advertiser? The answer is that an equally sophisticated and entirely parallel system exists for the advertisers to provide feedback to the intermediaries when you take an action that relates to an ad they showed you. That action might be clicking on the ad immediately, or it might be independently visiting that advertiser’s website in the future. (This delayed behavior is called “viewthrough attribution.”)
Toward the end of my visit, Mark Torrance demonstrated the remarkable precision with which his computers can predict and influence your behavior by showing me how he estimates the likelihood that you will buy a pizza from one of his clients (a major international fast-food pizza franchise) within two weeks after you see one of their ads. Selecting a green cell off an elaborately colored chart called a “heat map,” I could see that for a carefully selected group of consumers, between 9.125 and 11.345 percent of them would order a pizza from his client within two weeks, even without his client knowing whether they ever ate pizza at all. The actual figure, as later reported back to him by the client, was 10.9 percent.
The various participants in this arduous process aren’t exactly friends, and so all sorts of shenanigans and game playing ensue. For instance, because the winning bidder in any auction knows what price the runner-up bid, it can infer a lot about who else is out there gunning for the same inventory and what those other parties are willing to pay. So the bidding parties engage in complex strategies to outwit the other participants, like professional poker players sizing each other up by intentionally losing hands. And the ad exchanges’ synthetic intellects, which manage all the bidding, hardly behave as angels. They study each bidder’s strategies and increase their own profits by cherry-picking opportunities or pitting similar bidders against each other to run up the price.
With so much energy invested in this process, you might expect that these ads would be quite precious, but the opposite is true. Despite the Herculean effort these synthetic intellects put into each and every battle, the right to serve an ad through one of these ad exchanges might sell for as little as $0.00005, or five ten-thousandths of a penny. (In advertising terminology, this is a five-cent CPM—cost per thousand.) But, as the saying goes, they make it up on volume. Three friends founded Rocket Fuel in 2008 and, as of this writing, the company is worth approximately $2 billion. As you might suspect, both Mark Torrance and CEO George John studied artificial intelligence at Stanford.
So what’s the root cause of all this electronic pandemonium— computer programs fighting each other over the opportunity to game our financial systems or influence our consumer behavior? Can’t synthetic intellects just play nice, like decent civilized people?
The answer is surprisingly simple. These systems are designed to achieve singular goals, without awareness of or concern for any side effects. As I will explain in later chapters, there’s no incentive for combatants in these new electronic coliseums to show any mercy to each other, or to pay anything more than the bare minimum they must to get what they want. Similarly, they will charge the most they possibly can in order to extract the maximum profit possible.
As synthetic intellects increasingly encroach on areas previously the exclusive domain of humans, oblivious to the broader social context, they are prone to behaving in ways that society would find repugnant. Stealing a parking place that someone else is patiently waiting for; buying all the batteries on a Home Depot shelf before a big storm, leaving none for anyone else; perhaps blocking a wheelchair from using the curb ramp while waiting for the light to change.
But as these systems become ever more capable and autonomous, the danger grows dramatically. For instance, imagine a future in which someone buys the latest model of general-purpose robotic personal assistant and instructs it to apply its immense abilities toward becoming the world’s most successful chess player. The human may have in mind that the robot will study the games of grand masters, practice against other players, and enter various competitions. But without guidance, the robot may instead formulate more reliable strategies, such as threatening the families of credible competitors in the hope of throwing them off their game, sabotaging planes carrying better players to the contests, or otherwise incapacitating anyone who might interfere with its ability to meet its assigned goal.4
And what, if anything, should we do about the potential dangers posed by synthetic intellects? The answer is more nuanced. We need to control when and where synthetic intellects (or any electronic agent, for that matter) are permitted to act on our behalf. This need is particularly acute when they commingle with human agents.
I’ll start with this latter issue. We frequently rely on a hidden assumption of a level playing field to allocate resources in a reasonable way. When Ticketmaster first went online, it greatly increased the convenience of getting a ticket to a concert. (I’m old enough to remember having to drive to the nearest Tower Records for this purpose, where Ticketmaster located its specialized high-tech terminals. Actually, I’m old enough to remember the bad old days before Ticketmaster, when you simply went to the concert to stand in line and took your chances.) But soon after Ticketmaster was available on the Internet, scalpers began using programs to scarf up online concert tickets the moment they became available. Lacking a regulatory framework to address the problem, Ticketmaster has attempted technological fixes, such as requiring you to interpret those annoying little brain twisters known as CAPTCHAs, to little effect, because the scalpers simply employ armies of live humans, mostly in third world countries, to decode them.5
The problem here has nothing to do with whether you use an agent to purchase a ticket. It’s fine for you to buy a ticket on behalf of a friend or to pay someone else to do it. The issue arises when we permit electronic agents to compete for resources with human agents. In most circumstances, it violates our intuitive sense of fairness. That’s why there are separate tournaments for human and computer chess players. It’s also why allowing programs to trade securities alongside humans is problematic, though I think we would have a hard time putting that genie back in the bottle.
Lines and queues are great cultural equalizers because they force everyone to incur the cost of waiting, spending his or her own personal time. That’s why it somehow seems wrong when lobbyists pay people to stand in line for them at congressional hearings, squeezing ordinary citizens out of their chance to attend. Some argue that waiting in line extracts a higher price from the wealthy than the less fortunate, but that misses the point: there are some resources we don’t want to be economically fungible. It’s the reason it’s illegal to buy or sell votes, or kidneys, in most civilized countries.
This same principle, appropriately generalized, can apply to just about any circumstance where electronic agents compete with humans—not just to lines. Do the participants differ in their ability, or the cost they pay, to access the resource? This question needs to be answered on a case-by-case basis, but the concept is clear. For instance, suppose I send my robot to move my car every two hours to avoid a parking ticket, or instruct my self-driving car to repark itself. Will we judge that cost sufficiently equivalent to doing it myself to consider it fair to those without a robotic driver or car to spare? What if it costs me as much to send the robot as it would for you to send your human administrative assistant?
I contend that the brawl for the right to display an ad to you seems a lot fairer than having HFT programs participate in the securities markets. That’s because, in the case of ads, humans don’t typically participate in the placements (though they did in the early days of the Internet), so every bidder is on a more equal footing.
Once again, our biological baggage works against our interests. It’s easy to physically see if the boss’s robot is moving his or her car. It’s a lot harder to tell if someone has written a clever program to reserve an entire row of camping spots at Yellowstone Park the weekend you want to go the moment the sites are released to the public—while you’re still loading the web page. We need to incorporate these concepts into our public discussion so we can extend our sense of fairness into the electronic domain. Right now, it’s an untamed territory shrouded in perpetual darkness that invites all manner of skullduggery.
But there are much more subtle problems with the use of synthetic intellects as agents.