Chapter 4

RISK MEASUREMENT:

Hollywood’s Never-Ending Quest for Certainty

Il n’est pas certain que tout soit incertain. (It is not certain that everything is uncertain.)

—BLAISE PASCAL, PENSÉES

Risk-free is a single, predictable outcome. Risk is the opposite, all the things that might happen and the odds that they will. In a perfect world, we have a risk estimate that captures all possible outcomes and puts precise odds on each one happening. But the world is full of uncertainty, and we lack the imagination to anticipate everything that could go wrong (or right) and rarely know the exact odds of anything. All we can do is take a guess, and often the most scientific way to do so is to make a risk estimate: data from the past is analyzed and a range of things that might happen in the future is produced, along with estimates of the probability of their occurrence.

Sometimes it is easy to make an accurate estimate; at other times risk measurement is nearly impossible. When it comes to the challenges of measuring risk, I can think of no better example than the movie business. One of the hardest risk measurement problems that has eluded generations of risk modelers is putting a number on the odds that a movie will be a hit.

THE LAND OF BROKEN RISK MODELS

Hollywood is often called the land of broken dreams, and anywhere bets go so wrong so often is fertile ground for our risk exploration. Every day young, hopeful, talented people come to Hollywood hoping to make it big. But few realize these dreams and leave with bitterness and regret instead. Hollywood could also be called the land of broken risk models. Investors, including banks, hedge funds, and insurance companies, have a long history of coming to Hollywood thinking they can tame the market with science and data, which also often ends in tears or lawsuits. There is a saying in Hollywood financial circles: “The secret to making lots of money is come here with three times that amount.”

A recent casualty is Ryan Kavanaugh, a Los Angeles native who charmed Hollywood with talk of his Monte Carlo simulation* that lived in an elaborate Excel spreadsheet and promised to make the unpredictable predictable. He claimed his model could forecast which movies would do well and which ones would bomb. It was a seductive pitch.

Such predictability is seductive because it is so elusive in Hollywood. If past performance is any predictor of success, investors would have stayed far away, but everyone in Hollywood is looking for the next big thing in a sea of random outcomes. Like others before it, Kavanaugh’s model eventually failed, but not before many investors bought in.

People in the movie business explain that it is impossible to predict what will be a blockbuster or a flop. Each film is like a small business with hundreds of moving parts. The only way to manage risk is to make lots of movies; most won’t make money, but a few will hit it big and pay for the others. This is a risky way to run a business, and it also explains why there are so many bad movies, with terrible, derivative plots, that fail at the box office. Every year brings both a notorious clunker that cost hundreds of millions to make and an independent drama, with a great script, that cost only $10 million and earned $300 million.

This bet-on-’em-all strategy is a colossal waste of money and talent. Many great movies are never made while billions are squandered on bombs that people forget as soon as they leave the theater.

Predicting winners is an especially hard risk problem. In most businesses, decision makers can rely on data from the past to help them figure out the more fruitful investments that will pay off in the future. A good risk estimate requires data that can do two things: (1) reveal lessons from the past that will be relevant in the future, and (2) predict that certain past outcomes are more likely than others. The nature of moviemaking means its business data lacks both of those things.

To make matters worse, filmmaking is a particularly risky venture requiring a large up-front investment that doesn’t pay off for years, if it ever does. Studios scramble to reduce financial risk by securing financing from outsiders to take on the risk for them. Attracting these investors often involves piggybacking off the latest fad, signing a megastar to the project, or seeking the potential for merchandising revenue. These strategies are assumed to increase the odds of making money, but they don’t necessarily increase the odds that a movie will be good or even profitable.*

Investors who finance the films usually get equity, meaning they get a share of the film’s profits after writers, actors, directors, production crew, and editors are paid.* Because the expected return on most movies is less than zero, the investors shoulder most of the financial risk for next to no reward. To offset the risk, deals are often made for a slate of about a dozen movies at a time, but investors often cannot choose which films are included in the slate.

It seems baffling that anyone would agree to these terms, but investing in movies is exciting and sexy; you get to hang out with movie stars and go to film premieres. Matthew Lieberman, an executive at PricewaterhouseCoopers, says clients looking to get into movies are often sophisticated investors who become blinded by the glamour of Hollywood—attending award shows, hobnobbing with celebrities—and make investments they would never consider in other markets.

If someone could come up with a way to scientifically pick winners, then a well-functioning moviemaking market would be ripe for the taking. Enter Ryan Kavanaugh.

He grew up in Los Angeles as part of a privileged family and started a venture capital fund with his father after college that raised money from the biggest players in Hollywood to invest in start-ups during the 1990s. The firm fell apart after the dot-com bust in 2000, and Kavanaugh was sued by his investors.

Just a few years later he made a comeback and cofounded Relativity Media in 2004 before he turned thirty. Armed with a team of number crunchers, he marketed himself as a math whiz in jeans who could provide the predictability Hollywood and his investors craved. His timing couldn’t have been better because movie studios needed a new source of financing in the mid-2000s. For years they had depended on a German tax shelter to attract investors and off-load some of the considerable financial risk involved with making movies. But Angela Merkel’s coalition government nixed the shelter after she took office in 2005.

The German tax shelter had given investors and studios some financial incentive to invest in films, so losing it left studios unsure how to get financing. In the meantime, hedge funds were looking to invest in high-yield risky assets. It was a perfect match. Kavanaugh jumped on the opportunity, especially since hedge funds, with their roots in finance, have to put a number on any risk they take. He offered investors the two things they wanted. He gave them the glamour they craved. An entertainment lawyer who has worked with Kavanaugh told the New Yorker in 2012, “Ryan knows how to suck people into the glamour of Hollywood. You’re a banker, leading a dull life, and all of a sudden you’re hanging out with movie stars. You think, I’m walking down the beach with Gerard Butler! Before you know it, you’re rationalizing why you should be making this investment.”

And most critically, Kavanaugh claimed he could put a reliable number on risk, precisely what the institutional investors needed to hear before they put their clients’ money on the line to make movies. Kavanaugh would go to New York, visit the banks and hedge funds, talk the finance talk, and write equations on a whiteboard to put precise odds on whether or not a movie would make money.

The hedge fund managers needed this because measuring risk is what people in finance do. They feel more comfortable when they can put odds on success. We all do.

TURNING DATA INTO RISK: WHAT NORMALLY HAPPENS

No matter what type of decision you need to make—momentous or mundane—the simplest way to measure risk is to consider what happened in the past and assume something similar will happen in the future. This gives us a reliable estimate of the range of things that might occur.

If you drive to the same airport once a month, odds are it does not take exactly thirty-three minutes each time. More likely, it usually takes between twenty and forty minutes, depending on traffic or weather. That range doesn’t account for something unusual happening, like a terrible accident that causes an hour’s delay. In general, we make a decision based on the normal range of things that might happen. If we are prudent, we assume it will take forty minutes to get to the airport; if we can tolerate a little risk, we might only allow for thirty minutes.

Risk is our guess about what the future holds; more precisely, it is the range of things that might happen and how probable each event is. The odds of guessing a single outcome correctly—for example, that a movie will earn $200 million—is nearly impossible (even Kavanaugh did not promise he could do it), but the odds are we can figure out a range of possible outcomes that could happen. A summer blockbuster has an excellent chance of earning between $1 million and $4 billion at the American box office. Four billion dollars is possible, but it’s a long shot, and a summer release will almost certainly earn more than $1 million, so making a good risk assessment requires narrowing the range of possibilities.

You need a workable range for any risky decision. If you anticipated a three-hundred-car pileup whenever you drove to the airport, you’d always leave three hours in advance and in almost every instance end up wasting your valuable time sitting around an empty terminal.

The hard part is knowing what constitutes a reasonable range. Is twenty to forty minutes enough, or is traffic so unpredictable that you need fifty minutes or even three hours?

In financial economics, determining the ideal range is done a little more methodically. Estimating the range of what might happen using data is called risk measurement. Risk measurement, as we know it, is a recent human invention. Until the end of the Renaissance and the start of the Enlightenment, most people presumed uncertainty was determined by divine forces and couldn’t be measured. But in the seventeenth century the mathematicians Blaise Pascal and Pierre de Fermat started measuring probabilities for games involving dice. Their insight changed how scholars thought of risk: they started to view it as something that could be measured and controlled.

The mathematician Jakob Bernoulli took their contributions further about sixty years later, applying these burgeoning lessons to the real world, outside of the controlled situations with precise, quantifiable odds used thus far. He assumed the range of things that happened in the past could be used to predict the odds that something will happen in the future. One of his major contributions was the law of large numbers, which says if you repeat an experiment enough times, you can estimate accurate probabilities of what might happen in the future.

These pioneering statisticians provided the bedrock of modern statistics, the study of how we measure risk based on what happened in the past. For example, consider a stock price and how much it went up on down from one month to the next. The figure below shows how much stock prices, the S&P 500, went up or down each month between 1950 and 2018. Think of it as 824 trips to the airport, only for monthly stock returns. If you assume the future will be like the past, the figure on the previous page shows everything that could happen to the stock market in the next sixty-nine years and the odds that it will happen.

Notice the shape of this figure and how most stock returns cluster in toward the middle. In most months the stock market returns between −11 percent and 13 percent; a 16 percent return is very rare.

Financial economics often assumes the history of stock returns conforms to a certain shape called a normal distribution or bell curve. It is smooth and symmetrical, and most of the data is clustered in the middle. It looks like the figure below.

If you believe that the range of what can happen looks normal, you can make a quick estimate of risk. This is called a standard deviation, or volatility. Volatility tells you the range in which stock returns will vary most of the time. Or to be exact, in any given month, 68 percent of the time, the U.S. stock market will fall 3 percent or rise 5 percent, or some percentage in between. The bigger the range is, the riskier the stock portfolio (or any type of risk) is, because you can expect a wider possibility of things to typically happen. Investing in emerging-market stocks is riskier than investing in American stocks: prices will probably fall 8 percent or go up 9 percent, or some percentage in between.

If you were superneurotic about getting to the airport, you could use the same technique. Suppose you drove to the airport nine hundred times and estimate airport-travel volatility: the range of time it usually takes to get there is twenty to forty minutes. You’d also notice that a three-hour airport trip caused by a major traffic accident is less probable. It only happens 1–2 percent of the time. The traffic accident is called a “tail risk” because a three-hour trip is so unlikely it is in the tail of normal distribution.

These measurements are how people in finance define risk: they often assume a normal distribution and use volatility as the standard measure of risk. You can probably find a volatility estimate on your mutual fund statement. It tells you, roughly, how much you can expect the mutual fund to go up and down in price. It assumes a close to normal distribution. But it does not tell you much about tail risk, which, though improbable, could be a catastrophic outcome, like the stock market falling 40 percent.

Normality is a controversial assumption, and lots of evidence points to stock returns not conforming to this shape. If returns aren’t normal, then the range associated with volatility will understate the risk. So in our airport-travel example, a trip may take twenty to forty minutes only 50 percent of the time. Or the tail risk, the nightmare three-hour trip, may be more likely than you expect, with a three-hundred-car pileup happening 5 percent of the time.

Making movies in Hollywood is just like traffic: there is nothing normal about it.

THE MOVIE BUSINESS: SKEW YOU

Typically, it is hard to measure risk in the movie business because it is nearly impossible to pin down a reasonable range. A movie is like an airport trip that will take anywhere between ten minutes and two hours.

If you plot the history of movie profits, it looks totally different from the normal distribution shape assumed in finance.

The figure above shows the ratio of box office revenues (foreign and domestic) to production costs for all movies released and shown in at least one hundred U.S. theaters between 2008 and 2017. Any value less than 100 percent means ticket sales did not cover the costs of production. To cover marketing and additional costs not related to production, a good rule of thumb is that a movie must make double its production costs to be profitable.*

For decades, box office returns have had the same risk profile, despite the introductions of innovations like IMAX and competition from streaming and better-quality TVs. The economists Arthur De Vany and W. David Walls looked at box office receipts of 2,015 movies between 1985 and 1996 and plotted almost the exact same shape.

The figure is known as a skewed distribution. It is what makes the movie business the movie business. It also describes many decisions we face every day.

The asymmetric shape shows how risky and unpredictable the movie business is. If there’s a normal distribution and the center falls on breaking even, there are an equal number of profit-making and money-losing scenarios, and most movies fall within a close range of breaking even. With a positive-skewed distribution, like the previous one illustrating the movie business, the range of possibilities is large; there are far more profitable scenarios than money-losing ones. Notice the long tail on the right, which covers the range of potential positive profits. A movie in this range could barely break even or return more than 1000 percent, or anything in between. All the profitable scenarios are equally unlikely. The odds are a movie will lose money because most are clustered in the smaller, loss part of the curve. Fifty-three percent of movies shown don’t earn back production budgets at the box office, and that’s assuming they are shown in many theaters (most movies aren’t). And even if they do make a profit at the box office, their earning potential looks like a complete crapshoot with only a few big winners.

“Skewness” poses problems for measuring risk. For volatility to tell what will happen most of the time, you need a normal, symmetric distribution. If your distribution is skewed, volatility underestimates risk; it might only tell you what will happen 30–40 percent of the time. The long tail contains a huge range of possibilities; all are nearly equally probable and unlikely. Studios know that most movies will lose money, but that a few will be at the end of long tail and subsidize all the losers, but they don’t know which ones those will be or have any idea if they will be a mild or huge success.

It’s common for risk to be skewed; the symmetric distribution is called normal, but it is not always common. Aspiring movie stars who come to Hollywood face a positive-skewed distribution. Odds are they will never make it, but there is a large range of potential working-actor scenarios that have a small chance of happening, from landing regular bit parts in movies to becoming the next megastar.

Suppose you are thinking of leaving your steady, well-paying job to work at a new tech start-up. It pays less than your current position, but you get valuable stock options. Think through the range of bad things that could happen: the start-up might go bust or you might make less money for a few years and eventually leave. On the flip side, many good things could happen: the start-up could be the next Google and you’ll get rich; or maybe the company will be bought out and a nice windfall comes your way, but you still have to find another job; or maybe the start-up will grow into a bigger company and one day pay you what you earn now, but you’ll have more responsibility. While it looks like there are more good outcomes than bad ones, the bad outcomes are more probable because most start-ups will fail. If you plotted the range of things that could happen, it would look like a skewed distribution instead of a normal distribution. A bulk of the distribution is in the loss zone, but there is a long tail extending into all the successful but unlikely scenarios.

Actually, the investment strategies of venture capital firms, which put their money in start-ups, are similar to those of movie studios. Many of their investments lose money, but the odd unicorn pays off to make up for the losers. Kavanaugh’s history in venture capital was good preparation for convincing people to invest in long shots. The skewed distribution in this industry also explains why millions of dollars are poured into dud tech firms that are clearly a bad idea.

Kavanaugh claimed his model could generate a reliable estimate of risk and overcome the curse of the skew.

How did he do it? He selected certain movie characteristics (such as actor, director, genre, budget, release date, and rating) and estimated which ones would make a winner in the future by analyzing data for the same characteristics from previous films. The model produced a range of potential profits based on how these characteristics had performed in the past. Picking which movies to invest in based on certain factors can mean less risk because the distribution such a strategy produces can offer more reliable risk estimates.

For example, action movies are riskier investments because they are more expensive to make. From 2008 to 2016, the average production budget for an action movie was about $104 million versus a more modest $19 million for the average horror movie. Only about 35 percent of action movies earned back their production costs at the box office compared with 67 percent of horror movies. So Hollywood makes more horror movies, right? Wrong. Between 2007 and 2016, more than two times as many action films than horror films were produced (216 versus 103).

The figure on the following page plots the range of payoffs for both action and horror movies. More action movies get made for many reasons: they tend to do well internationally; they offer the possibility of franchising and merchandising; and because their box office returns are less skewed, their performance is more predictable and they are therefore less risky as investments. Horror movie returns, on the other hand, have a very long tail: many lose money, and there’s a wide range of payoffs for the winners. Even if they are profitable more often than action films, they are in some ways riskier because they are less predictable.

Kavanaugh claimed that his model could produce a range of reliable potential earning scenarios, because the process of selecting certain characteristics* enhanced predictability and the odds of earning a profit. And if more than 70 percent of those earning scenarios were associated with enough profit, Kavanaugh told investors to put their money in the film as part of a slate of other movies he handpicked. Studios were so enthusiastic about the potential for financing that they shared full data on their profits with Kavanaugh; he called it the “Holy Grail” of Hollywood.

The Excel spreadsheet contained the Holy Grail data and transformed it into something even more elusive and desirable: reliable risk estimates that the hedge funds and banks needed to green-light investment. They plowed hundreds of millions of dollars into the movies Kavanaugh picked. In 2005 and 2006, he financed thirty-six movies with Universal and Sony and made money for his investors. Hedge fund investors earned a $150 million profit on one of his early slates, a return of between 13 percent and 18 percent. Kavanaugh was paid millions of dollars per movie and got a producer credit despite having no role in production.

But then Kavanaugh got greedy. Elliott Management, a $21 billion hedge fund, paid $67 million for 49.5 percent of Relativity in 2008. This gave Kavanaugh access to the money he needed to start investing in movies himself. His spending got out of control: his private bathroom had toilet paper with an image of President Obama on it; he brought exotic animals into the office; and he started to work out of a lavishly decorated airport hangar. Even worse, his magic model stopped working, selecting bombs like The Warrior’s Way, which cost $42 million to make and brought in $5.7 million in the United States, and Machine Gun Preacher, with U.S. earnings of only $539,000. Elliott Management pulled out in 2010. Kavanaugh managed to find more financial backers, but he continued to struggle as his spending accelerated and he picked more duds. Relativity was bankrupt by 2016.

Once again, Hollywood broke a risk model.

THE PAST IS A LOUSY WAY TO PREDICT THE FUTURE

Another reason measuring risk in Hollywood is precarious is that the data gets stale fast. Kavanaugh’s infamous Monte Carlo simulation predicted the future, but the inputs his model relied on were from the past.

And for a while it worked. Investors got the numbers they needed to feel comfortable, and they made money off their investments. It seemed as though Kavanaugh’s simulation did what no other model could do. But that’s the thing about predicting the future based on the past. It works until it doesn’t, because the market (especially for movies) keeps changing, and estimates based on old data no longer tell you much of anything; what is difficult is knowing when you need to update your data. Often we don’t realize the world is changing until long after it has changed.

In the last ten years alone, DVD sales dried up, China became a bigger market, and franchise movies about comic-book characters became more profitable. Streaming and better TVs mean people are less inclined to go to theaters. Online review sites like Rotten Tomatoes can undermine even the best-laid marketing plans. It has led some industry experts, like the Wall Street Journal reporter Ben Fritz, to argue that the market has fundamentally changed forever. That would mean data from fifteen years ago tells us nothing about the movie market today. He argues studios will make fewer movies each year going forward and concentrate on films about comic-book characters.

Stale data undermines more than predicting which movies will break box office records. Voting patterns from the Obama/Romney election weren’t relevant for making inferences in the Trump/Clinton election, which resulted in misleading poll projections. Technology and more global trade alter old economic relationships and make past data less relevant today.

The JPMorgan media executive David Shaheen described Kavanaugh’s model as “garbage in garbage out.” He argues that Relativity Media used the wrong data, the wrong way. Maintaining a data set that is accurate and can pick which movies are winners is difficult if not impossible when the data changes so quickly. Shaheen and his colleagues speculate that while comic-book franchise movies look like a sure bet today, the market will eventually become saturated and another fad will come along. Unpredictability means Hollywood quickly latches on to fads one day only to totally abandon them the next.

TERRIBLE IS THE BEST WE’VE GOT, AND IT’S COMING TO A THEATER NEAR YOU

Kavanaugh overpromised because there are no perfect risk estimates. Risk, a measurement of uncertainty, is a human construct that attempts to bring order to an unknowable future. Risk is meant to help us understand what we are up against and plan for what might happen, good or bad. It also helps us weigh different options and see which ones bring us closer to our goals. We use data to make choices every day: trying out a new restaurant because we enjoyed the chef’s food before or returning to our favorite resort because we had a great vacation there last year. Sometimes these estimates fall short, because management at the resort changes or the chef’s new restaurant is not as good.

Data may be a terrible way to predict the future, but it is the best we’ve got because it is all we have. The limitations of data are in some ways becoming more apparent in a rapidly changing world that renders past data useless in an instant. At the same time, data is becoming a more powerful tool to measure risk. The modern world is taking the original ideas of Pascal, Fermat, and Bernoulli even further because now we have more and better data, with more computing power to measure risk than ever before. Endless amounts of data exist on what we buy, what we watch, and who we know. We have apps on our phones that can turn this data into predictions about likely flight delays, how well we’ll match with a blind date, and stock market ups and downs.

More data and estimation techniques, like machine learning, mean more reliable risk estimates. Soon things that once seemed immeasurable, like the odds a movie will be successful, could be possible.

Netflix can give you a recommendation based on the odds that someone with your demographic profile finished viewing a movie. Instead of making risky decisions based on a rough estimate from your past moviegoing experience, you can make decisions based on millions of other people’s experiences. As Bernoulli showed, more data means more accuracy. That will empower us to make more informed decisions, though we also need to be aware of data’s limitations.

A question remains that data cannot answer yet: Does Hollywood make so many bad films because it is so hard to measure risk with changing data and skewed distributions, or does a poorly functioning, undisciplined market in which the biggest risk takers, the people who finance the movies, don’t reap the biggest reward but invest anyway because of the glamour, create the skewed distribution?

We’ll soon find out.

Technology is again changing the movie market, given the rise of people streaming content in their homes. Amazon and Netflix, which are now in the production business, have data on precisely who watches what and whether they finish. Right now, almost half of a movie’s budget is spent on marketing because films are advertised to everyone, with the vague hope the advertising will appeal to someone. Now that studios have data on viewing patterns, they can better tailor their marketing strategy and know which movies will appeal to their intended audience for far less money. This is expected to change the distribution of potential outcomes, narrowing it and making it more predictable.

This might transform the kinds of films that are made, reduce the skew, and maybe even means we’ll start seeing better movies.