News Feed, Lurking Variables, the Confirmation Bias, and the Principle of Correlation and Causation
A good plan fails because it doesn’t adapt to the changing environment. Those in charge of the strategy are separated from those in charge of the implementation and thus unaware of the modifications being made in the field or unaware that the tactics aren’t really working. Plans need modification because of the fallacies of forecasting. Adaptive management is the process of making adjustments that are aligned with the strategy and adjustments that make the business model work. Adaptive management is designed to fuse the planning process with the implementation process. And at the core of the implementation process is a methodology called “management by metrics.”
The inception of Plan A involved the creation of metrics. Since our business model is merely a means for solving a complex hierarchy of problems, the metrics we chose to track are merely a reflection of those problems. Our tactics, meanwhile, are the things we do to solve the problems and drive the metrics. If we do this right, management by metrics creates a feedback mechanism that allows us to make good business decisions that are consistent with our strategy and the problems we’re trying to solve.
In 2006, Facebook was just beginning to take off and Mark Zuckerberg was conceiving new tactics for the site. One in particular had him up late at night and excited about the prospect of revolutionary change. “It’s not a new feature,” he said, “it’s a major product evolution.” It excited him because it was based on the core problem he was trying to crack and the underlying theory he had about solving it. To Zuckerberg, Facebook was a communication platform, a means of enhancing existing relationships, and a way to become more open and honest with your friends and family. This meant that the “time spent on the site” (TSOTS) was the primary metric he chose to watch. If he created a feature that increased the TSOST, the feature was deemed successful. If it decreased TSOST, it wasn’t.
Zuckerberg also studied click-through paths and the way members used the site. Doing this, he observed that a huge amount of time was spent “surfing” the sites of social networks looking for changes, new photos, and updates. So he decided to automate this process. He created a feature that automatically searched for such changes and pushed them onto the user’s Wall. He called it “News Feed.” He imagined that everyone was a celebrity, that on the Web everyone had their fifteen minutes of fame, and News Feed was like People magazine, which kept you updated on the minutiae of life the way we follow Britney Spears or Paris Hilton. He considered News Feed the culmination of his understanding of social networking.
In September, Facebook launched News Feed. It was a complete disaster. Almost instantly, the feedback was negative. A few minutes after launch, the company was inundated with angry e-mails. The first one read, “Turn this shit off!” Within hours protest groups began organizing on the site, an online petition was started, a boycott was called, and ironically, a Facebook page dedicated to killing News Feed was created and garnered almost a million friends within a few days. Zuckerberg was alarmed and confused. His team begged him to turn it off, consider it a mistake, and get back to basics. It would have been an easy thing to do. The customer is always right, after all.
But that’s not what happened.
You see, News Feed was a tactic firmly rooted in Zuckerberg’s hypothesis about social networking. People were complaining about privacy, but there was nothing that News Feed was pushing that wasn’t already openly available on the site. People thought that it bordered on “stalking” and called it “creepy.” But it was just automating what people were already doing. That’s why the reaction confused Zuckerberg so much. Shawn Fanning, one of his business advisers, said, “If it didn’t work, it confounded his whole theory about why people were interested in Facebook.” So instead of just abandoning it, Zuckerberg studied it.
At first his team was shocked that he didn’t take it down. But he was managing by metrics. Sure, people were complaining, but the TSOTS had increased dramatically at the launch of News Feed. In other words, his primary metric told him that the tactic was successful; people were spending more time on the site than ever. The protest groups were able to organize so quickly because of precisely the feature they were protesting against. In his mind it was a success.
Zuckerberg wrote an open letter to Facebook users. “We really messed this one up,” he said. “When we launched News Feed and Mini-Feed we were trying to provide you with a stream of information about your social world. Instead, we did a bad job of explaining what the new features were and an even worse job of giving you control of them. I’d like to try to correct those errors now.” And to his credit he did. He added more privacy controls and simplified the updates, and after the initial protest and a few more iterations, users began to embrace it. It was a turning point for Facebook and a turning point for its leader. A lesser manager would have caved in. But a lesser manager would never have had such a strong connection between the underlying theory of the problem he was solving and how it manifested itself in the tactics of the business model. Zuckerberg was able to make bold business decisions because they were rooted in theory and he carefully chose the right metrics to manage by.
Metrics are the things we measure to test the effectiveness of our tactics. And tactics, well, they’re what doing business is all about. In business, warfare, science, and mountaineering, tactics reign supreme.
The Supremacy of Tactics
A strategic plan alone isn’t worth the paper it is written on. Its value lies in the ability it gives you to make good business decisions. We do specific things—tactics—that are designed to create certain outcomes—metrics—and this creates a direct relationship between tactics and metrics. The day-to-day operations of a business are mostly tactical in nature. We are constantly trying to find ways to drive our leading metrics, and the tactics we create are the ways we do it. With every new tactic we learn something about our business, and these lessons drive our understanding and the evolution of the business.
Here’s an important truth, the maxim that all strategic thinkers must realize and the one strategic concept most commonly forgotten: when strategic plans fail, they fail at the tactical level. H&R Block’s foray into financial services failed in the stores, on the front line, and in the minds of customers. Block never developed any successful cross-selling tactics. Likewise, the failure of the space shuttle program was the result of failed tactics; NASA was never able to build a spacecraft with reusable, redundant, fail-safe parts. The shuttle was not a truck; it was a sophisticated, complex launch vehicle with a high failure rate built in. Both Block and the shuttle, however, are strategic failures. Both failed to solve the problem they had identified, but the failure happened on the tactical level. You see, tactics will always trump strategy. Without successful tactics, a strategy is useless. On the other hand, an organization built upon a successful tactic is going to be successful, at least in the short term, as long as the tactic continues to work.
With this understanding, it’s rather absurd how little time is given in most large organizations, at the top levels, to the creation, monitoring, and evaluation of tactics. Leaders see themselves as pathfinders, their primary responsibility as guides giving direction to the organization. Though this is certainly true, leaders often forget that their plans will fail without the right tactics. They say that it is the responsibility of mid-and low-level employees to develop tactics—in other words, to implement the strategies and directions. Some feel that playing in the tactical space is micromanagement and beyond their responsibility. But they are, in effect, blindly delegating the most critical function of the organization. All Fortune 500 companies have a vice president of strategic development. He or she usually reports directly to the CEO. However, none, as far as I know, has a vice president of tactics. Tactics are lost. It’s like a mountaineering guide leading a certain route but unconcerned with the team’s tactical equipment, like crampons, ropes, food, shelter, water, and fuel. It’s like a general ordering a flanking attack but not giving his troops guns and ammunition. I’ve seen less of this in smaller organizations simply because they live and die by their tactics; their leaders are on the front line and so are well aware of the tactical situation. Put simply, great strategic companies are great tactical companies.
There’s no doubt that Mark Zuckerberg is a brilliant strategic thinker. He’s able to see and sense the long term and lead his organization through the strategic mire of the complex world of social networking. But—and this is a big “but”—he is very much a tactical thinker, and the history of Facebook is the history of a company obsessed with tactics, with fine-tuning existing features and having the courage to add new ones. Early features of Facebook included the profile, notes, status updates, groups, and Wall. The Wall was an important feature; it allowed users to post information about themselves and for friends to leave notes and comments. It’s what made Facebook so hypnotic in the first place. But not all of the features were strategic; some were just tactics for tactics’ sake (which is a true bottom-up thinking move). Take, for example, the “poke” feature. About this early feature, Zuckerberg said, “When we created the poke, we thought it would be cool to have a feature without any specific purpose. People interpret the poke in many different ways, and we encourage you to come up with your own meanings.” Today the poke has evolved, and it’s used to say “hello” to friends and as a type of flirting for others (when I was younger, “poke” had a very specific meaning, like “poking” someone with a pencil or a…. well, you get the idea).
Of course, not all Facebook features have been successful. In 2007, Facebook added a virtual gift feature. You could send small icons and novelty items, such as Valentine messages or charitable donations, for a dollar. It lasted a few years and never really took off.
Management by metrics is how we determine the effectiveness of our tactics. In online marketing, there’s a concept called “site optimization,” which is a process used to drive the evolution of a website by testing different elements of the site to see which one produces the intended result. It is a pure form of management by metrics and is really a form of tactical optimization.
Site optimization is a concept borrowed from direct-mail marketing in which a company improves its marketing program by testing variations of it against a control or winning design. New design tactics, aimed at optimizing a certain behavior such as “unit volume,” are developed and then tested against the existing ones. We call this A/B testing. For example, imagine that you have a gray “Buy Now” button in a 12-point font on your site. You theorize that a bigger button may be more noticeable and increase your sales, so you make a 16-point button and test it against the control button (the 12-point button). You do this by sending 5 percent of your customers to the test page and the other 95 percent to the control page. To your delight, the 16-point button beats the smaller one, so you now make it the control button. You reinforce success and abandon failure by getting rid of the smaller button. But you’re not done; if a 16-point button is better than a 12-point one, what about an 18-point button? Could it beat the 16-point one? You run the test, and again, it does. So then you make a 20-point button, run the test, but find that it doesn’t beat the 18-point one; you sell less product. So you’ve just discovered that 18 points is the optimum size for your “Buy Now” button. You then proceed to do the same thing with the other elements of your design, such as price, product, colors, and offers. Through such systematic testing, over time, you’ll develop the best winning combination of tactics that will together optimally drive your business model. This is the process that Facebook uses to optimize its site.
In developing new tactics, you should do so based on your theoretical understanding of your business, test them against your best ones, and then get rid of those that don’t work as well. This isn’t a process that you can use just with e-commerce sites or direct marketing, it’s a process that all business models can (and should) employ. The evolution of the Apple iPod is a good example of such optimization. For the first few years, the iPod got smaller and smaller. The “small” version became the Shuffle, and it was popularly used by runners and people working out. It solved the problem of entertainment while exercising. Over time, it got smaller and smaller and became a standard piece of gym equipment. But then the Shuffle got too small—so small, in fact, that it became difficult to control. The product died, and just recently Apple abandoned the “too small” version and went back to the previous one. This is just like the “Buy Now” button getting too big; there’s an optimum execution of your tactics, and you’re constantly in search of it. The iPod is a thing in motion. It’s difficult to beat Apple, it’s difficult to emulate them, because they’re constantly changing, they’re a moving target. This is the result of adaptive management.
You need to make sure that your business is also always searching for new tactics. In fact, I believe that most large organizations should establish the position of vice president of tactical development. This person would have the responsibilities to search for new tactics and to lead the optimization process.
Reading Your Metrics
When reading a metric, there are three possible results: an increase in effectiveness, a decrease in effectiveness, or no change in effectiveness. As adaptive managers, we’re looking for either an increase or a decrease. In other words, we’re looking for tactics that move the needle. No change in effectiveness means the tactic is useless, while an increase or decrease means we are affecting the business model. Let’s say, for example, that we work for H&R Block and want to drive more foot traffic into its stores because walk-ins have been decreasing over the last few years (the problem). A root cause of this problem could be that the stores, are old and rundown and so people are not enticed to walk in. A hypothesis for solving it would be to invest to give the perception of a modern, upscale, vibrant tax office. So we decide to invest $20,000 in new signage outside one of our tax stores (the tactic). We would measure the increase or decrease in foot traffic (the metric) with the new sign. One of three different things can happen: an increase in traffic, a decrease in traffic, or no change.
If there is no increase or decrease, we can conclude that the sign has little effect on foot traffic (the tactic doesn’t work). We’d also note that we neither proved or disproved our hypothesis. It could be that our hypothesis is true but the tactic is ineffective. We may want to search for another tactic or just abandon our hypothesis. We learn nothing when there is no change in a metric from a tactical implementation.
Now let’s assume that there is an increase in foot traffic. This means that our tactic is effective and we have proven that our hypothesis is true. We could then reinforce our success, for example, by creating an even more “modern, upscale, and vibrant” sign to see if we can improve the metric even more (tactical optimization). Since we’ve learned that modern is good, we can also reinforce success by updating the interior design to make it more “modern, upscale, and vibrant.” In other words, we would begin searching for more tactics to reinforce our hypothesis. When a metric increases, we learn that the tactic works, but we don’t really know why. It may be because of our hypothesis, or it may be some other reason. (We will talk about “other reasons” later in this chapter.)
Finally, let’s assume that there’s a decrease in foot traffic. We put up a modern sign, and fewer people come into the store. What does this mean? First, it means that our tactic is effective; we’ve just executed it the wrong way. We’ve learned that signage is important. Second, we’ve learned that our hypothesis is wrong. For some reason, for H&R Block prospects, a “modern, upscale” look doesn’t work. We could then theorize why. Perhaps, for example, it is projecting an image that gives the impression to prospects that the service is going to be very expensive and so they don’t bother to go in. We would then design a new sign that projects a more value-based image and test that one. This is an important lesson in managing by metrics and one that is counterintuitive and often overlooked: you learn more from a failed test than you do from a successful one. As an engineer, I don’t find this counterintuitive. We learn more from a bridge failure than we do from one that remains standing. When a bridge fails, it reveals its weak point. When a business test fails, it reveals the weakness in your hypothesis. This is why, as an adaptive manager, I’m excited when a tactic fails badly.
You see, management by metrics is the search for cause and effect. And cause and effect are the primary goals of your business model. You are employed to make things happen—to create effects.
The Principle of Correlation and Causation
Adaptive management is based on the scientific method, which, in turn, is based on the Principle of Correlation and Causation. We’re searching for cause and effect, and we use correlation to find it. However, we can be misled by lurking variables and faulty data; correlation doesn’t always mean causation.
Cause and effect are so fundamental to your thinking process, always working, always calculating, that they’re like water is to a fish, so abundant that you don’t even notice they’re there. It’s how we learn, how we adapt, and, ultimately, how we make predictions. We do it so well that Charles Darwin said that it’s what sets our species apart from the lower forms of life. Like other cognitive skills, causal reasoning begins in infancy and develops gradually as new experiences accumulate. When my daughter, Katie, was younger, I remember her building numerous if-then connections as she began to learn and explore her world. During dinner she would turn her glass of milk upside down, giggling (along with me) as it splashed across the floor. Essentially, she was working out the cause-and-effect laws of gravity as well as the cause-and-effect laws of Mom and Dad. Things fall if they’re not held up; moms get mad over spilled milk; and dads seem to think it’s quite humorous. Three causal beliefs were confirmed, yet my poor daughter still hadn’t figured out if it was right or wrong to spill milk.
The physicist Carl Sagan once said, “Science is a way of thinking much more than it is a body of knowledge.” That way of thinking, as we discussed earlier, is based on cause and effect. A scientific hypothesis is a guess at the cause of an effect. What’s called “Newton’s curse” is that we perceive the world as deterministic, one thing causing the next, which causes the next, which ultimately leads to the effects of the present moment. It’s so fundamental to our psychology that it tends to be ignored.
Business history shows us how technology changes things through causation. For some innovations, there’s a ripple effect, a deterministic consequence, one thing leading to the next. Consider three of the great inventions of the last one hundred years: the lightbulb, the automobile, and the Internet. Sure, each of these things created its own industries, but it also changed other industries, destroyed some existing ones, and created completely new ones. The electric lightbulb created the power industry, but it destroyed the gas lamp business. The power grid, which was built to service the lightbulb, was then used to power other things, like electric ovens, electric brooms (vacuum cleaners), and electric heat. The automobile created a new system of paved roads, but it destroyed the buggy whip business. The roads, in turn, led to the invention of the suburbs. That changed the way people shopped: the local grocer was replaced by the supermarket. Now people could buy things in volume because they could transport their purchases in their Model Ts instead of having to carry them. Today we see the same thing happening with the Internet. It was created so that universities could share data and information, but it was hijacked by business. Now we can buy books or schedule an airline flight at home from our computer. But it destroyed travel agencies. Then social networks were created, giving us Facebook and Twitter and changing the way we interacted with one another. One damn thing after another.
How do we do determine cause and effect? We use a mathematical tool that was created by a nineteenth-century Victorian gentleman.
Sir Francis Galton was a Renaissance man in the same mold as his cousin Charles Darwin. Among other things, he was an explorer, geographer, inventor, meteorologist, and gifted statistician. A prolific intellect, he wrote more than 340 papers and books. He studied the inherited characteristics of sweet peas, and in the process, to better understand them, he developed the concept of linear regression. This led to the mathematical concept of correlation. Correlation’s been a boon for statistics ever since, making it immensely practical for forecasting. It’s also been a bust, for it’s deeply misunderstood and used incorrectly every day in business and life in general.
Correlation is how we measure the relationship between two variables. Once it is understood, we use it to make a prediction about one variable based on what we know about the other variable (or variables). Galton, along with Karl Pearson, developed a formula to calculate correlation in terms of coefficients. These can range from -1 to +1, and they tell us how closely things are tied together.
A positive correlation means that if one variable increases, the value of the second variable increases, too. For example, research has showed us that there’s a positive correlation between income and education. As the number of years of education increases, so does average income. People with higher incomes tend to be more educated than those with lower incomes. The closer to 1, the more highly correlated the two variables. Income is highly correlated to education because the coefficient is 0.79. Cause is tied to effect. So we could use the correlation coefficient to predict how much money someone is going to make if we know how many years of education he or she has. We could then hypothesize that more education causes people to be smarter and the effect is that they can make more money.
A negative correlation, on the other hand, means that if one variable increases, the other decreases. For example, there’s a negative correlation between years of education and years spent in jail. People who have more years of education tend to spend fewer years in jail. Negative correlations are inverse correlations and tend to be just as telling as positive ones. It’s when the coefficient is closer to 0 that we have no correlation and so little or no chance that we’re seeing a cause and effect.
Correlation is what helps us determine causation, which, in turn, helps us to make predictions and manage by metrics. But correlation can be misleading because it doesn’t always mean causation.
Lurking Variables
In Minnesota, there’s a strong correlation between ice cream sales and drowning. It’s alarming. As ice cream sales increase, the rate of drowning deaths increases sharply, with a correlation of greater than 0.8. Evidently, eating Ben & Jerry’s and then going for a swim is a deadly combination. In Florida, a study showed that there’s a direct correlation between the use of artificial sweeteners in place of sugar and weight gain. People who use more NutraSweet are more likely to be overweight than those who don’t. And an international study cited by the National Cable & Telecommunications Association showed that there’s a direct correlation between the number of television sets in the home and life expectancy. It seems that watching TV makes you live longer. And you thought American Idol was useless drivel!
However, before you buy another TV, you’d better look more closely at these correlations. Each of them is an example of misleading connections and lurking variables. In other words, just because there’s a correlation between A and B, it doesn’t mean that A causes B. In fact, whenever there’s a correlation there are three possible things happening:
A causes B
B causes A
or
C causes both A and B
Lurking variables are things behind the scenes that cause effects but are unknown to us at the time we’re looking. When C causes both A and B, we call C a lurking variable. Ice cream sales and drowning are highly correlated, not through causation but through the lurking variable called temperature. As it gets hotter outside, this causes both an increase in ice cream sales and an increase in the number of people who go swimming (which increases the number of drownings). The correlation between NutraSweet and weight gain isn’t that NutraSweet causes weight gain, it’s the result of a lurking variable called “dieting,” and people who diet are more likely to use NutraSweet and gain weight. And finally, watching TV doesn’t increase life expectancy; there’s a lurking variable called “wealth”—people who have more money buy more televisions and also have access to better health care, chlorinated water, and better nutrition. People in Angola and Bangladesh don’t have TVs in every room.
In business, we run into the same problem with lurking variables. When I was consulting for an insurance company and we were trying to devise a membership retention strategy, we noticed a high correlation between retention and the use of the company’s website. Great, we thought, if we can drive our members online, they’ll be more likely to stay with us and not choose a competing insurance provider. Makes sense, right? However, before we implemented this new and expensive tactic, we did some statistical detective work, sensing that there might be a lurking variable. You see, we knew that people who made claims were more apt to stay with the same insurance because they’d established a relationship and felt that jumping to a new provider would make their rates go up. Perhaps this was skewing our data. So we removed the members who had made claims over the last twelve months and then looked again at the online user retention rates. The claimants had a nearly 100 percent retention rate, while the others had a standard retention rate. There was no correlation between retention rate and online usage unless you were making claims to your policy. Needless to say, that made us rethink our strategy. I often wonder how many times I’ve developed a new tactic based on lurking variables and the misunderstanding of correlation.
As Randall Munroe, a computer programmer and popular blogger, said, “Correlation does not imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look over there.’”
Before we leave this chapter, there’s a final bias we need to understand as we analyze our data and make adjustments to our model. It’s inherent in the way we think, and it can cloud your perception of the metrics, especially if we’re working with a business model whose metrics are difficult to measure.
The human mind is wired to detect patterns. We’re very good at it; so good, in fact, that we imagine patterns even when they don’t really exist. This means that we all have the inclination to validate our beliefs and hypotheses. Researchers call this the confirmation bias, and it is so prevalent that it makes us overconfident in our beliefs and can lead to denial and bad decision making. It’s also partially to blame for why we tend to stick to Plan A instead of evolving into Plan B. It creates an unrealistic devotion to our original strategy. In business, it manifests itself in three ways: how we collect information; how we interpret information; and how we selectively recall this information from memory.
The confirmation bias is the result, in part, of how information is gathered and stored in the mind: first in, first out. Initial impressions tend to be the most significant, and they are the thoughts we want to defend. We remember earlier information better than we remember later information. That’s why some people find it difficult to overcome childhood traumas and why our earliest influences are often the greatest. It’s very difficult to change someone’s mind once it’s made up. According to Jonathan Baron, a psychology professor at the University of Pennsylvania, his experiments have shown that initial ideas and information make a much greater impression than subsequent ideas and information. He says that you’ll naturally form a more positive impression of someone if he or she is described as “intelligent, industrious, impulsive, critical, stubborn, and envious” than as “envious, stubborn, critical, impulsive, industrious, and intelligent.” Reversing the order of the description changes the impression we get, leaving a much stronger memory trace. Once we have the impression, we’ll work to defend its position in the mind, to confirm it.
According to Thomas Kida, the author of the book Don’t Believe Everything You Think, confirming data, as opposed to refuting it, is a fundamental cognitive strategy that most of us employ. He calls this the positive test strategy and says that it’s the natural way that we think. In other words, we tend to “think of yeses instead of noes when we consider a certain issue.” In a thought experiment conducted with a number of students, he used the following sequence of numbers to explain how this works:
2 4 6
These numbers obey a certain rule, he said, and it’s your objective to determine what the rule is. In order to decode the sequence, you can choose other groups of three numbers and you’ll be told whether your groupings obey the rule or not. According to Kida, the thinkers will come up with a hypothesis, such as “even numbers increasing by two.” The subjects then give a sequence like 12, 14, 16—a grouping to confirm their hypothesis. Yes, they’re told, that sequence obeys the rule. So they pick another grouping, like 50, 52, 54—and again they’re told that this sequence, too, obeys the rule. They’ll usually try a few more confirming sequences and then confidently exclaim, “I know the rule, it’s even numbers increasing by two!” They’re disappointed when they’re told no, that’s not the rule.
The rule that Kida was looking for was “any three numbers in increasing order.” The positive test strategy, as you can see, is not an effective way to determine this rule. A thinker can come up with thousands of examples of “even numbers increasing by two,” and each one will confirm the real rule. What most fail to do, according to Kida, is to disprove the hypothesis. For example, we could have given 7, 9, 11, and when told the sequence obeys the rule, we would have realized that the “even numbers” part of our hypothesis was wrong. However, very few people ever try to disprove their hypothesis. It’s not the way we typically approach a problem.
This is true in business. Leaders are constantly looking for data and even manipulating the data to prove that their market theories are right. Very few spend the time or resources trying to disprove their understandings, so they wallow in denial while they point at confirming data from studies, statistics, and anecdotal information. Later, we’ll study Roberto Goizueta of the Coca-Cola Company, who spent millions of dollars confirming his theory that the new formula for Coke was better than the original formula and would be more popular than the existing one. In fact, there were plenty of data during the tests to refute the viability of New Coke even before its launch, but Goizueta and company tended to overlook them or even alter the tests. For example, during the early focus groups for New Coke, a few participants in each group became agitated when they discovered that the company was going to change its formula and said they would stop drinking Coke if it did. The researchers told Goizueta that those people tended to skew the results toward the negative because they exerted peer pressure on the other participants (a typical focus group phenomenon). So the researchers were told to get rid of those negative participants, to screen them out before they had the chance to exert peer pressure. Essentially, they changed the way they collected the data so that they could confirm the superiority of the new cola formula. (I’ve seen the same thing happen in other focus groups. Company executives ask for certain participants to be pulled for the exact same reason, to confirm their hypothesis.) Incredibly, even after the debacle and the rerelease of Classic Coke, even after New Coke garnered less than 3 percent market share, Goizueta continued to drink New Coke until the day he died, convinced that it was a better cola. It wasn’t until five years after his death that the company discontinued New Coke production.
It’s not only how we collect information but how we interpret it that exposes the confirmation bias. When I was the president of Preferred Capital, I believed that customers preferred having a finance company separate from the company that sold them equipment. My entire business was based on this premise, so I was constantly searching for information to confirm my brilliant hypothesis. I remember, one time, looking at a report that showed that 80 percent of our customers were offered financing by the equipment vendor but chose to use our financing. See, I said, that just confirms my theory. “No, it doesn’t,” one of my managers replied—because he had done his own research. We were closing only 10 percent of the deals we were working on. He had done a survey of the other 90 percent and found that an overwhelming percentage of them were financing with the company that sold the equipment. The reason, he told me, was that it was “more convenient,” which was the exact position in customers’ minds that we were trying to occupy. We had both looked at the same piece of information; for me it confirmed my hypothesis, for him it refuted it.
Three Stanford researchers conducted a series of experiments that documented this thinking bias. Charles Lord, Lee Ross, and Mark Lepper had about a hundred students fill out a questionnaire that solicited, among other things, their views on capital punishment. A month later, those with strong views either for or against capital punishment were asked to participate in an experiment as part of a course requirement. Half of the participants were for capital punishment and thought that it had a deterrent effect. The other half were against and thought it had no deterrent consequences. Then each group was given a detailed study that contained both “pro” and “anti” data. When asked about the data, the pro–capital punishment participants were quick to point to the “pro” information, while the anti–capital punishment participants talked about the data that supported their view. Nothing surprising there. Neither group was swayed, and each weighted the information that confirmed their beliefs. However, when asked to comment on certain “pro” information, the pro participants said things such as, “The experiment was well thought out, the data collected were valid, and they were able to come up with responses to all criticisms,” while the anti participants said things such as “I don’t feel such a straightforward conclusion can be made from the data collected.” When confronted with contradictory data, the participants discounted the data itself. At the end of the study, not one participant had changed his or her viewpoint, even though they had all been exposed to data that clearly refuted their positions. In other words, they all used the data to confirm their beliefs; not one of them used the data to refute them.
Finally, and perhaps more important, we all have the tendency to remember those things (be they data, statistics, studies, or even anecdotes) that support our viewpoint. This is called selective recall, and it’s the final manifestation of the confirmation bias. Even if we’re exposed to information that contradicts our hypothesis, we’ll tend to only remember the data that confirms it. Even though my manager at Preferred Capital had raised doubts about my theory of separation between the equipment vendor and the finance company, I kept remembering the customers who had said they liked it even though they were probably a small minority. I kept using these encounters to trump any other data and let them confirm my beloved theory.
Management by metrics is relatively easy to do with an online business model because it’s easy to define the metrics and easy to measure them, and the feedback is instantaneous. Other businesses aren’t so deterministic. For a business model like that of Facebook or an online candle business, which provides clear feedback, reading metrics is relatively straightforward and close to the scientific method. For other business models, such as that of the space shuttle or Coca-Cola, it becomes more difficult and more of an art than a science. The methodology, however, remains the same. You can do it, but these biases get in your way. Now that you understand them, though, you can strive to compensate for them.
Before we go there, though, we have to explore the two primary ways we can alter our business models. This is, perhaps, the most important chapter in the book.