“Bouncy” high… but low… speech. Using expressions like “OK?,”With the year 2000 approaching, countries around the world began considering how they might celebrate and commemorate this essentially arbitrary numerical event. In the UK an initial concept for a “World’s Fair” showcase event was developed by Tony Blair’s government into something much grander: someone hit on the idea of building the largest single-roofed structure in the world and putting some stuff in it that signified Who We Are, What We Do, and Where We Live. But at the end of its year-long life the Millennium Dome was widely regarded as a flop.
In 12 months, 6.5 million people visited an attraction that had cost the country more than £600 million.1 This was 5.5 million people fewer than the original estimate and almost 20% below the “worst-case” scenario estimate provided by consultants Deloitte & Touche, who factored in “risk factors” like the possibility of the content being insufficiently attractive and marketing failing to pull in the forecast mix of visitors.2 A subsequent review by the advertising agency M&C Saatchi entitled “Will 12 Million Visit the Dome?”, based on NOP tracking research, concluded that the figure of 12 million was “conservative,” given a pool of between 16 and 18 million people who had made statements that caused the research company to classify them as “likely” or “persuadable” visitors.3
To confuse matters somewhat, research conducted during the year the Dome was open found that 87% of visitors were “satisfied” with their visit, and 86% were “satisfied” with the services provided by the Dome’s hosts; another survey found that “nearly all respondents were aware of the Dome.”4
So where had it all gone wrong? The moment the organizers asked people about the Dome, they were on a slippery slope toward a 6 million person fall – because they were asking the wrong people.
I’m not suggesting that the research sampling was awry, rather that the people questioned by the research process had little or no connection with the ones who would ultimately make the purchase decision. To some extent this is a by-product of the issues raised in the previous chapters: someone’s decision to visit the Dome is a balance of unconscious influence and the context at the time the decision is made. For example, when the idea was raised in research several years before it was due to take place, there was growing excitement about so many digits on the calendar changing simultaneously. But once January 1st had passed, the Y2K computer problem hadn’t spawned Armageddon, and everyone had adjusted to a 1 and three 9s turning into a 2 and three 0s, life settled back into its familiar patterns. Making a trip down to a relatively inaccessible borough of a busy capital city with a poor transport infrastructure, to an “event” that was being regularly criticized (and no one was claiming really contained the “greatest show on earth” as the government had promised),5 didn’t seem like quite such an attractive proposition.
However, this wasn’t solely a matter of not appreciating the Dome in its actual context. Additional surveys conducted by NOP in April and August 2000 suggested that 15 million and 12.4 million people “had already visited, are likely to visit or could be persuaded to visit.”6 With the folly of the Dome publically exposed, why were people still saying they would go when, as the year-end figures proved, they weren’t going to? The answer is that the research interview process does more than merely ignore critical components of why people behave as they do, it changes how and what they think.
In a world where so much time and money is spent on market research in one form or another, there is every chance that, at some stage, a statistic or report will be pushed in front of you as justification for a decision. Concern about the quality of research tends to be focused on the validity of the sample and the statistical significance of any differences in the data, but as the Dome’s experience illustrates, the statistical methodology can be pure and the results still grossly misleading. There are 13 reasons most questions should be avoided, which will be explored in this chapter.
Raising something as a question pushes it into the conscious mind for a conscious response. It frequently also makes a presumption about how relevant or interesting that issue is to the person concerned. In an understandable attempt to explore what someone thinks about something, the very fact that you ask them about that thing is a potential distortion of reality. For example, in asking how trustworthy I consider a brand (or yourself) to be, you presuppose that, at the moment of decision making, trustworthiness is an influential variable in the decision.
This point was illustrated by one of my clients when they asked me to help them understand whether their investment in a new store design was worthwhile. When they had tested a new store previously, they had used “accompanied shops”7 to get customer feedback. The research had told them that the new store was a significant improvement and was very much liked by customers. However, when more stores were refitted and could be assessed in terms of their sales performance against the rest of the estate, the client could see no evidence of a financial return to justify the additional investment.
Unfortunately, the researchers’ questioning during the accompanied shops had prompted customers to consider various aspects of the store, not least those in which the company had invested and on which they were keen for the research company to report back. These questions presupposed the importance and, indeed, existence of the new elements in the customers’ experiences; as soon as the researcher asked about an element of the store, it was reasonable for the respondent to examine it, consciously appraise it, and reply.
I discovered that many of the new elements contained in the store went unnoticed by customers, and that key elements of the store that were unconsciously referenced during the retail experience had not been changed. In particular, customers scanned the perimeter of the store to navigate it, enabling them to retain one focal length as they scanned the environment, and ignored relatively large features that had been created in the middle of the store, but that were not relevant to their visit and didn’t require referencing for navigation.
While this phenomenon can be hard for people to accept when they not only know what is there, but have taken the decision to invest a considerable amount of money to have it put there, it is well documented in psychology. Studies such as those by Simons and Chabris have proved that people often ignore apparently significant visual events if their attention is focused elsewhere (even a gorilla walking into a group of basketball players and beating his chest can be overlooked through so-called inattentional blindness).8
When you become conscious of other people’s extraordinary capacity to fail to notice things, you become more aware of when you do the same yourself; though on the basis of my own experience, no better at combating it. Shortly after writing this passage I was searching eBay for a particular brand of golf club that I’m interested in buying. A seller had listed exactly the clubs I was interested in and I clicked on the listing to read more. There was relatively little information provided, just one picture and only four lines of text, amounting to thirteen words in a font much larger than the one that you’re reading now. In addition the seller, who also operated an eBay shop selling unrelated products, had added a disclaimer that the golf clubs were a personal sale and, reassuringly, not a martial arts weapon.
After reviewing the information, I sent a short email to the seller to ask what condition the clubs were in; the picture resolution was too low to tell and I was a little concerned that this most basic information wasn’t included. After sending the email I returned to the listing and, for some reason, looked at the information again (it is shown below exactly as it appeared in the advert):
Tour Edge Bazooka Iron-Wood
Reactive Flex Regular
5-9 PW SW
In excellent condition
I emailed the seller and apologized for my inattentional blindness!
It doesn’t help the accuracy of research that, once they’ve agreed to take part, people are almost always helpful enough to answer the questions they’re asked. However, all the responses provided are not the result of equal thought or awareness of the issue concerned. When people were asked if they thought the US government should spend money on an antimissile shield, the results appeared fairly conclusive: 64% thought the country should and only 6% were unsure. But when the pollsters simply added the ambivalent option in the question “…or are you unsure?” the level of uncertainty leaped from 6% to 33%! When they drilled down marginally below the surface and asked whether respondents would be upset if the government took the opposite course of action from their preferred route, 59% of people either didn’t have an opinion or didn’t mind if the government did something different; a far less compelling endorsement for spending all that money.9
The fact is that asking about something overrides the natural state that thing occupies in someone’s experience. It’s very hard to preempt what people will find interesting or attention worthy – which makes it very risky to presume by asking them a question about it. When research has put a focus on the issue it’s investigating that causes people to consider it a way they otherwise wouldn’t, it has manufactured the response it gets.
The way in which questions change mental processing doesn’t only undermine the process of asking other people questions. Similar problems can arise when asking yourself a question about consumer experience that can lead you to arrive at erroneous conclusions, something that was demonstrated recently when I was observing customers to evaluate the impact of a new store design.
Over the course of a couple of days at the store (and from previous work for the client) I had had the opportunity to observe a large number of customers shopping there. While there were some important but subtle differences in response to the environment, for the most part people behaved in a very similar way. From my observation and subsequent interviews with a sample of them, I knew that people were usually in a very unreceptive mindset when they arrived, screening out large areas of the store and focusing (at least initially) on the specific product they had come to view. The store was located on a retail park and there was very little “passing trade” or browsing, since shoppers had to put a reasonably large degree of planned, conscious thought into the decision to travel there. Where the store design was successful, it did draw people into other product areas and entice them to look at and interact with other items, and I knew which these points were and how the location and design of the fixture and nature of the products had influenced this customer behavior.
On the second morning, two men walked into the store and behaved very differently. They started an unplanned but nevertheless systematic sweep of the whole store. They looked at everything: the signs, the lighting, and the carpets. It was easy to conclude that these people were not there to look in the store, they were there to look at it. Knowing of the intense rivalry that existed between my client and its main competitor, and on the basis of their appearance, I guessed that the men were senior managers from the competing retailer. I decided to try to “interview” them at the end of their visit.
When I asked if they would take part in some research, there was a momentary pause during which I suspect that the more senior of the two was weighing the chance of him learning things from me about his competitor against his irritation of taking part in the research. His professional curiosity won through and he agreed to answer some questions. His responses confirmed what I had already observed: he had appraised the store in a totally different way from a genuine consumer. He used commercial category terms and made comparative references to other stores around the country. He talked about lighting levels, signage, and the flow of the store, and he aesthetically appraised the new fixtures. Real consumers don’t talk about any of these things and don’t make any conscious evaluation of them during their purchase experience. Indeed, they are oblivious to most of them at a conscious level.
Unsurprisingly (at least to someone versed in the principles of transactional analysis), the conclusions the men came to based on their conscious, balanced (adult) assessment of the store’s design had nothing in common with the reaction of real consumers. In many ways, as soon as they asked themselves the perfectly reasonable question “I wonder what that new store is like?” and decided to go and look in order to answer it, they were destined to misinterpret what they saw, at least in terms of how a consumer might have directly or indirectly appraised it. Consumers had a very different perspective, being preoccupied with conscious matters (“Have they got what I want?” and “Where is the thing I’m considering buying?”) and unconscious matters (“Do I feel happy and safe here?” and “Do I feel in control?”). The two men had not seen the store as consumers saw it, to the extent that they may just as well have gone and looked at a different store!
There is also evidence that simply asking people questions about something will change the answers they provide. Wilson and Schooler designed an experiment in which people were given a number of jams to taste that ranged in quality (on the basis of an expert taste panel) and asked to rate them. Some participants were asked for their reasons for liking or disliking each jam before rating it; others were given an irrelevant questionnaire first. It transpired that the people who consciously deconstructed their jam preferences devised criteria that were unlike those used by the experts, based their ratings on these reasons, and came to a different conclusion about which was best. In contrast, the group who weren’t encouraged to think about the taste of the jam in this abstract and artificial way had preferences that corresponded very closely with the expert panel.10
We like the notion that our judgment is self-contained; it is after all our own judgment and it will be what it will be. However, research shows that our judgment is far more malleable than we might like to believe. Tormala, Petty, and Clarkson asked participants about their perceptions of a fictitious store that had been described to them in terms of three of its departments.11 The store description was always the same, but the information participants were exposed to immediately prior to seeing it varied. What emerged was that, whether the information was about a competing store, a car, or a hypothetical person, the nature of that information affected the responses to the description of the department store. When the first message was sketchier and less informative, the department store was perceived more positively.
It is very easy to demonstrate just how influenced people can be by contextual information. If you ask someone to think of a number they will quite often say 7, and very often say a number between 1 and 10 (because that’s a common mental association with the suggestion). However, if you ask someone to think of a number and first tell them that you yourself are thinking of the number 876, it’s likely that they will come up with a three- or four-digit number of their own. You can demonstrate how apparently irrelevant information primes people in a similar fashion. Contrive a reason for someone to think of a larger number, for example by talking about how wide the Atlantic is, and then ask them to think of any number. For a linguistic example, you’ve probably heard the brain teaser that primes someone with a detailed story about a fatal plane crash involving people of two different nationalities that comes down on a border and asks in which country the survivors would be buried. Everyone knows you don’t bury survivors. However, when the answer isn’t obvious the unconscious is just as ready to latch on to any peripheral information available at the time and link a response to it, irrespective of its relevance.
Salespeople know that they can prime reactions to the price of their product by using larger numbers before they get to their “bargain” price. Similarly, consumers will flock to a discounted product, even without any absolute knowledge of the usual undis-counted price of the item.12 When people have been thinking in terms of numbers at one numerical level, they use that as a base point from which to define another number.
It’s also possible to be primed by the most trivial suggestion. While taking part in a charity rally with a friend of mine, I was bemoaning the fact that I had forgotten to buy new toothpaste and would have to eke out the remnants of a tube I had considered long since dead until we passed a supermarket or pharmacy. My friend echoed my concern, commenting that he only had a tiny tube supplied in a wash kit from an airline to last him the five days of our journey from the UK to Portugal; he had hoped the razor and toothpaste he’d picked up would last the trip, but now he wasn’t sure. The following day it transpired that he hadn’t got any toothpaste at all. Primed by a notion of what other airlines included in the travel kit and, presumably, the nature of the tube, he had brushed his teeth as planned. He thought it wasn’t particularly minty or tasty toothpaste, but put that down to the airline’s poor choice of supplier. In the morning, his mouth soured by a particularly unpleasant taste, he checked the tube again, only to discover that it contained shaving cream.
It seems that it is impossible to prevent this priming effect. When Kahneman and Tversky did their ground-breaking work on behavioral decision theory, they put numbers around numerical answers, for example from a wheel numbered from 1 to 100 that appeared to spin at random (in fact they were controlling the outcome), and observed that people’s subsequent responses to questions with a numerical answer were influenced by the number to which the wheel had spun.13 When Timothy Wilson asked people to guess the number of physicians in a phonebook, he offered a substantial prize, warned one subset of the participants that people could be influenced by numbers they’d seen in earlier questions when making estimates, and urged them to be as accurate as possible. Even under such conditions, when people might be expected to draw on all their rational resources, the estimates provided were influenced by irrelevant numbers placed in the preceding questions.14
How big a difference can priming make in surveys? David W. Moore, author of The Opinion Makers and a former senior editor at the Gallup polling organization, compared two polls looking at US citizens’ support for oil drilling in Alaska’s wildlife refuge. One found that the public was opposed to drilling there by a margin of 17 percentage points. The other, conducted within a month of the first, found people in favor of drilling there by exactly the same margin. (Both polls corresponded with the interests of the groups that had commissioned them.) The poll that found more people in favor of drilling preceded that question with 13 others about the cost of oil and the country’s dependence on foreign suppliers. The poll that found more people against asked only the question on drilling in that region of Alaska.15
Psychologists have also found that the way in which people make evaluations about one set of products changes how they evaluate subsequent products. When a process has encouraged people to think about similarities or differences between brands, as is the case in a “market mapping”16 exercise, it can fundamentally change how they think about another, unrelated product.17 The way in which people think about the second product is, at least in part, defined by the previous exercise.
Similarly, several studies have found that the unconscious impact of one product, or series of products, influences the response to a subsequent product from an entirely different category. The same is true in response to advertising: where people looked at a basic product after seeing a premium one, they found it more appealing than when it was viewed in isolation.18
Recently researchers experimented by manipulating the difficulty of an article on movie reviews from a film festival that people were asked to read before looking at an advert for a watch. They discovered that if the article was difficult to read (something they manipulated by the type and size of font selected rather than the slightly more subjective adjustment of the content itself), people responded more favorably to a subsequent advert that was easy to process. It seems that, in the context of the difficult article, the positive relief of being able to understand the ad easily was inadvertently projected onto the advert. The research also discovered that when the advert had a connection to the content of the article, people liked the ad less even if it was easier to read. It seems that the unconscious associations between the company and the difficult article caused the reader to like the advertised product less too.19
It is eminently reasonable that research should want to make comparisons, be it of competing brands, different product formulations, or packaging options between which a company is choosing, or to get a relative position against its competition. After all, knowing that your company is “trusted” (whatever that might mean) by 65% of its target consumers is not as interesting as knowing that your main competitor is trusted by only 41% of people. To achieve this, research has to present respondents with a number of different alternatives and ask whatever questions it considers interesting. Unfortunately, by imposing a number of options for the benefit of the research design, such research is creating an artificial dimension that can affect the responses it receives.
It would be good to think that whether we like something is a discrete and independent matter related only to our tastes; after all, in a sense we define ourselves by our choices and we may believe that we alone are in control of these. However, psychologists have found that the range and nature of choice affect what people choose and how they feel about it. People given a wider selection of chocolates to choose between found the ones they ended up selecting less tasty, enjoyable, and satisfying than those given a more limited choice.20
The same issue arises in the question of what someone will choose from a number of options. When researchers asked people to choose between two cameras, a Minolta X-370 priced at $169.99 and a Minolta Maxxum 3000i at $239.99, 50% chose the X-370. However, when all they did was add in a third option, the higher-priced 7000i at $469.99, the proportion choosing the X-370 more than halved to 21%.21 In another study, the addition of a second choice of CD player over a stand-alone product resulted in the proportion of people deciding to purchase a product dropping from 66% to 54%.22
In another study, three quarters of doctors presented with a new drug for osteoarthritis would prescribe it rather than refer patients to a specialist. When a second drug was presented as an alternative option, the proportion deciding to refer patients to the specialist increased significantly. The choice we make doesn’t exist as an absolute, it’s dependent on the number of alternatives available.
People are inherently open to suggestion. I’m not talking about the “hold up a security van at gunpoint,” Derren Brown-style suggestion (although a small proportion of the population is that suggestible), I’m referring to the way in which we all unconsciously filter what’s going on around us and feel a particular way as a result. For example, if someone asks you how your life will be different in five years’ time, you could think about any aspect of your day-to-day existence and speculate on how it might change. However, if someone asks you how your life will be different but includes a prompt or two, perhaps by asking whether you will you be living in the same house and doing the same job, the probability that you will talk about accommodation and work is extremely high.23 While it’s unlikely (although not impossible) that such a question would be used by a good market researcher, this example illustrates an issue that can manifest itself far more subtly.
The unconscious mind, preoccupied as it is with rapidly processing and filtering lots of pieces of information, references countless aspects of our environment, including what it hears, and conditions us according to what it finds. A particular word or phrase triggers a set of associations: we get the unconscious reaction first and then consciously make sense of it (this is one of the reasons we’re able to communicate so rapidly, and without knowing what we’re going to say in advance). The consequence is that our decisions and responses become a by-product of what’s been said and are not fixed personal values at all. Psychologists refer to this issue as framing and it doesn’t just influence what we think in abstract terms, it influences what we do.
In one experiment, doctors, patients, and students were asked to choose between two forms of treatment therapy for lung cancer. They were given survival data on the efficacy of surgery and radiation; one group was given information on the probability of living, and one on the probability of dying. When the information was framed to tell them that people opting for surgery had a 68% probability of surviving beyond one year, surgery was chosen 75% of the time. However, when the question framed the data on the basis of mortality (i.e., 32% will be dead within one year) only 58% chose the surgical option.24 Another study, which asked people to decide who got custody of a child in a divorce based on short descriptions of each parent, showed that the answer shifted significantly depending on whether the question asked who they would “award custody to” or who they would “deny custody to”; the simple change in wording swung the majority from one parent to the other.25 Opinion polls have been found to be extremely sensitive to the choice of language used. For example, a poll is more likely to show public support for something when it frames the question as the government “not allowing” it rather than “forbidding” it.26
Unfortunately, problems are caused by not framing a question too. When research is asked in an abstract way it will lead to a different result from when more information is provided; it’s easy to be in favor of something when you haven’t considered the true cost and when the question doesn’t prime you to consider it. Four polls asking about renewing and expanding the US State Children’s Health Insurance Program (SCHIP) found support ranging from just 40% (52% were against) to 81%; the differences appeared to stem from different levels of explanation about the program, whether it was made apparent that there was a political divide in support for the bill, and whether the cost was mentioned.27
It’s worth noting that the price of tickets to visit the Dome weren’t announced until March 1999, two years after the research used to estimate visitor numbers.28 However, even when allowances are made for people not having known the price of entry, the nature and quality of the attractions, and the critical reaction of the media, research conducted in August 2000 (by which point the Dome had been open for seven months) still put 12.4 million people in the classification “already visited, are likely to visit or could be persuaded to visit.”29
As much as researchers might like to believe that they are being dispassionate (and as I mentioned previously, it’s questionable whether many actually achieve a genuinely noninfluencing questioning style), there is a very fine line between showing or describing to someone something that you want them to talk to you about and promoting it in their mind.
There is strong evidence to show that people are significantly influenced by how vividly information is presented. For example, when researchers presented a Save the Children appeal for African famine victims referring to the scale of the problem in statistical terms of the millions of people affected, only half as much money was donated as when the problem was depicted in terms of its effect on just one little girl.30
In a research setting, the focus on the subject matter created by being asked to stand and think about a particular issue, or by being presented with a description of a brand or product to appraise, is inherently artificial. However, it may also present the information in a way that shapes how people respond to it. The more detail you give respondents to consider, the more they will say. But in providing a vivid depiction of the brand, product, or service (or conceivably even when asking respondents questions that require them to construct such a depiction in their mind), another source of inadvertent influence has been introduced and the consumer responding is another step away from the one who will be making purchase decisions in the real world. It’s easy to conceive of the way in which respondents created a vivid mental image of the Dome’s “greatest show on earth” and convinced themselves that they would simply have to go and see it when it opened. Nevertheless, despite spending £40 million, the agencies tasked with attracting visitors were unable to recreate the same level of desire through their marketing efforts.31
Where the research has deconstructed or packaged the subject matter in an artificial way, it is impossible for it to be an accurate reflection of what people really think or do.
It’s very common for qualitative research to ask what people like about a particular product. Any research appraisal of a product being developed is likely to ask a consumer, or group of consumers, their opinion of the product. Either overtly or because it is an automatic basis for conscious evaluation of something new, people talk about what they like and dislike in what they are being shown. While this may seem inherently balanced, since both the positive and negative are being sought, there is a risk that in searching for the positive and postulating it, respondents unconsciously alter their position favorably.
You might think that beliefs are inherently stable. People will go a long way for what they believe in; some people will even wrap themselves in explosives and die for their beliefs. Certainly, market research has been interested in beliefs for years and considers them the foundation that underpins attitudes. Questionnaires frequently include attitude statements with which respondents can agree or disagree; this provides a way of getting responses that go beyond the monosyllabic limitations of quantitative surveys, without incurring the cost of using open-ended questions that will later require categorizing into meaningful groups and proportions. Such questions are “wonderful” for researchers, because the necessarily loaded nature of the attitude statements ensures data on something respondents would probably not express in a structured survey.
However, social psychologists have shown that asking someone to talk about something can change their opinion about the subject matter. Janis and King found that rather than their being fixed, beliefs can be created through behavior. Participants who made a speech playing the role of someone who believed in a particular issue were found to have become believers in the issue itself afterwards.32 In other words, the act of making the speech formed the “belief,” rather than a prior belief being constant throughout the forced experience.
So when researchers ask “What do you think the company is saying about its product in this ad?”, they don’t realize that the process of respondents conceptualizing the message of the communication may predispose them to accept it.
More recent research by Shen and Wyer found that simply asking people to choose whether they would buy or reject each of a number of products encouraged respondents to search for favorable attributes before unfavorable ones, and resulted in them regarding the next product they saw more favorably than they otherwise would have done; the question itself primed people to think more positively.33 Ultimately, the process of asking someone to evaluate something can change how they actually feel about it, or how they feel about another thing that you talk to them about subsequently.
Blind testing is a good example of a research technique that is commonly deployed to provide an “unbiased” understanding of how well a product performs in the mind of the people who might buy it while simultaneously failing to appreciate how those minds really work. The notion that a product tested without branding is somehow being more objectively appraised is entirely misguided (as Coca Cola discovered to its cost with New Coke). In the real world, we no more appraise things with our eyes closed and holding our nose than we do by ignoring the brand that is stamped on the product we purchase, the look and feel of the box it comes in, or the price being asked.
Our reliance on brands isn’t an indication of some form of shallowness or lack of intelligence; it’s a pragmatic system of packaging up product associations into an unconsciously identifiable device that removes the need to make complex and long-winded conscious evaluations of alternatives every time we purchase something. The fact that the system isn’t perfect, and that we may end up buying a product that we had associated with one set of attributes that it turns out not to have, doesn’t invalidate the approach most of the time, nor provide any practical alternative. A customer faced with hundreds of choices needs a way to filter what’s available; the unconscious mental power we (usually) no longer need to guard against life-threatening animal attacks can be deployed to helps us get a decent tin of beans quickly.
Many of us would like to think that we aren’t so shallow as to be unduly influenced by what’s written on something, but recent research suggests that we’re far more influenced than we might think. Researchers from Duke University found that even when brand logos were shown subliminally (flashed at speeds beneath conscious awareness), participants’ subsequent behavior was changed in a way that reflected the established values of those brands. Comparing responses to creativity tasks after subliminally exposing people to either an Apple or an IBM logo revealed that people who had seen the former brand, associated with nonconformity, innovation, and creativity, devised more unusual and creative uses for an everyday object (in this case a brick). They conducted another test using the Disney Channel logo and the one for the E! channel34 and found that people primed by exposure to Disney behaved much more honestly in subsequent tests.35 Since none of these people knew they’d seen the logos involved, they couldn’t have been consciously influenced by them and couldn’t have explained their subsequent behavior in the tasks; as far as they were concerned, they were simply being themselves and behaving as they believe they typically would.
In functional magnetic resonance imaging (fMRI) research, German radiologists have found that brands can change the way people think. When studying the brain activity of volunteers as they were shown different brands (car manufacturers and insurers were used) and asked to answer basic attitudinal questions about them, the researchers discovered that strong brands were processed with less effort and activated areas of the brain involved in emotional processing and associated with self-identification and rewards.36 When the impact of what’s on the product is considered in this light, it seems optimistic, to say the least, that anything relevant to a product’s sales performance would be learned from testing it “blind.”
Mindful of people’s capacity to fabricate answers, I tested how unreliable they can be when I worked on a brand and product development project for a biscuit manufacturer. I asked people to taste a new product (without showing them the pack) and tell me what they thought of it. There was widespread approval for the product, with a high proportion of the people interviewed claiming they would buy it if it was available where they currently bought one of the company’s existing varieties. However, I was confident that launching the product would not lead to successful sales for the company. In fact it had already been on the market, stocked right next to the products the respondents had been purchasing regularly, for several years; it transpired later that several of the people I spoke to had in fact bought the product in the past. In this instance the brand’s packaging didn’t encourage customers to have a high regard for the taste of the product – a perception advertising had inadvertently reinforced – and, unless the brand undertook a dramatic packaging redesign, perceptions of the taste of the product wouldn’t change.
Another temptation facing brand owners is to explore their brand in isolation from its product. When Lever Fabergé, part of the Unilever group, wanted to build on the success of its Lynx deodorant, it believed that research had given a clear understanding of the brand. The company was convinced that young men were attracted to the personality of the brand, rather than the brand as a deodorant.37 It decided that Lynx could be extended into a chain of barbershops. It furnished the salons with all the things it knew appealed to young men, like gaming consoles and MTV, stocked them with Lynx products, and designed them to look “butch.”38 However, after just 14 months the project was scrapped and the salons closed; they had failed to meet their sales targets. Lever Fabergé had made the mistake of looking at just one part of the puzzle and believing what people said.
Similarly, drawing attention to an aspect of a product that seems logically relevant may well create an artificial focus on something the consumer would not consider, at least not in the way they would when forced to by an interviewer’s question. When a New York liquor importer was considering importing a Swedish vodka to the US in the late 1970s, he decided to explore the potential for the product by spending over $80,000 on market research. The results were compellingly negative: people weren’t interested in a Swedish vodka; some people didn’t even know where Sweden was. However, the president of Carillon Importers didn’t like the idea of having wasted $80,000, so decided to see if the company could sell $80,000 of vodka anyway.39 Three decades later, over 70 million litres of Absolut are imported into the US each year.
A few years ago I had the opportunity to watch a research agency that specialized in packaging design research moderating some focus groups for a drinks company. At the outset, the person running the group spent a large amount of time facilitating a discussion about how the respondents currently used the brand concerned. Essentially, the group conducted a mini brainstorm and eventually dredged up a large number of brand references from advertising and packaging. Among these was the shape of the bottle and the occasion on which they purchased the product.
The brand’s big problem was that people only bought it once a year at Christmas. Unfortunately, the upshot of a process that forced people to appreciate that they only bought this particular product, in its particular bottle, at one time of year, was that the respondents consciously appreciated what they currently did unconsciously and therefore constructed justifications for why that would be the case. Consequently, when a series of innovative packaging designs was introduced for consideration, respondents were heavily (and unnaturally) sensitized to their existing behavior and were quick to dismiss something new; to have done so would have been to suggest that how they currently purchased the product was somehow “wrong,” when they had just rehearsed with themselves and each other why it was “right.” While there were numerous other reasons for this particular research approach being flawed, the fundamental problem was that the questions asked at the start of the interview had inadvertently set the tone for subsequent answers.
Most qualitative researchers conduct a “warm-up” exercise at the start of research (be it individual interviews or group discussions) to establish rapport and encourage the respondent(s) to talk openly. Unfortunately, this apparently unrelated exercise will prime people to bring particular thoughts or experiences to mind, which then color responses to subsequent questions.
Imagine that you’re taking part in research and are asked: “Where did you buy your last pair of shoes?” and “Why did you buy them from the place you did?” You’ve just publically declared some no doubt sensible reasons for buying your shoes from wherever you bought them, a shoe shop for instance. I’m sure you could talk about your shoe purchase easily (assuming it was reasonably recent) and that, having started talking in my presence, you will feel inclined to continue doing so, even if I start to ask slightly more challenging questions. Leaving aside the previously highlighted issue that your answers will be erroneous conscious post-rationalizations of what was probably a partially unconscious experience, I’ve just sensitized you to a process that was almost certainly not consciously constructed in this way when it originally occurred. If I now introduce a whole new concept of shoe buying, how likely would you be to embrace it? After all, we’ve both just heard your very sensible reasons for using the shop you chose (and probably both for the first time, too).
It is relatively common for research to explore consumer attitudes to brands, products, or services. A whole raft of possible thoughts relating to a brand is devised, and research respondents are asked to say to what extent they can identify with the sentiment the statements contain; often an attitudinal scale is used so that people can indicate the degree to which they agree or disagree. There is a widespread acceptance that if you can identify someone’s attitude to something then you have information that is indicative of how they will behave. Undoubtedly, this notion is attractive since it is how most people would prefer to believe they themselves function. If someone likes brand X the most, it seems logical that all things being equal, they will select brand X. Of course, things are rarely equal, and if the unconscious mind isn’t filtering by likability there is no reason for the outcome to reflect that dimension.
As far back as 1934 Richard LaPiere discovered that claimed attitudes to racial prejudice didn’t reflect behavior.40 He visited more than 200 hotels and restaurants with a Chinese couple and found that only one refused to serve them. When he wrote to ask the policy of the establishment six months later, over 90% claimed they would not serve Chinese people. Subsequent studies have found virtually no correlation between attitudes and behavior across a wide range of subjects. For example, you may not be surprised to hear that a lot of people around the world have “green” attitudes but show little or no evidence of environmentally friendly behavior.41 Similarly, we may want to believe that we like something (healthy food, for example), but an analysis of our past purchases (or waistline) may reveal that less healthy selections are made far more frequently.
In research I conducted as part of my undergraduate thesis, I explored the attitudes of 11–16-year-old schoolchildren to statistics. At the time a diverse set of subjects made some use of statistical methods and someone who ran my statistics degree course must have been interested to know whether this was destined to produce a generation of students with a passion for their subject. As part of this review I devised a questionnaire containing, among other things, a battery of attitudinal questions. The technically correct approach to such questions is to include balanced pairs statements, so that if one presents an issue negatively, another (some way down the list) will present it positively. What I discovered was that the children tended to agree with whatever statement they were answering, effectively contradicting themselves. I hypothesized at the time that they were too suggestible to use attitudinal questions reliably. I have since come to realize that several factors, not least the environment, could have contributed to this (the interviews were conducted in a school), and that the statements themselves were not likely to be a reliable indicator of anything in any case!
In the 1950s, Eric Berne developed a concept of how people interact, observing that the way in which they did so varied depending on the nature of the “transaction” taking place. While this is only one aspect of the psychoanalytic theory of transactional analysis that he developed – the basic “Parent—Adult—Child” model of personality (sometimes referred to as the first-order structural diagram) is the best known, the arguably more useful functional and structural variants less so – it has far-reaching implications for the accuracy of research.
What Berne realized was that there were distinct packages of thoughts, feelings, and behaviors that any one person could exhibit in response to the nature of the interpersonal exchange taking place at the time.42 In other words, how someone is spoken to can cause them to think and behave differently.
I find that ego states – or, as I prefer to think of them, “frames of mind” – are easiest to understand through observing other people. One of the best examples I have experienced was at a Seniors Tennis event at the Royal Albert Hall. Having watched a hugely entertaining and closely contested match from very close to the action (I was seated in the front row, virtually next to a line judge), I decided on impulse to see if I could get my program signed for my young tennis-playing son by one of the most famous tennis players of all time, John McEnroe. The layout of the venue provides a great opportunity to intercept the players as they leave the auditorium: they pass through the circle seats and walk through the public corridor running around the perimeter. In my Free Child mindset I was excited and happy; I’d really enjoyed the match and wanted a token of that to share the following morning with my son, who at 4 was too young to attend the event. Just three of us waited for McEnroe to intercept him as he passed through: me, another man about my age, and a young boy, of perhaps 8 or 9 years of age. From the appearance of the others I’m sure they were as excited as me – our thoughts, feelings, and behavior were very much aligned, we were in the same ego state.
There was a problem, however. McEnroe had lost his match to Paul Haarhuis and he wasn’t happy. He was angry, very angry. As he stomped by us the other two held out their programs. I could see trouble brewing and stood back, slipping my own program and pen behind my back. McEnroe went into a Controlling Parent mode43 and pushed past the man, muttering angrily, a mostly “inner” voice berating himself for having lost the match. However, when he saw the child he switched briefly into a Nurturing Parent mode. He appeared happy to vent his anger at losing the match at the man, but at an unconscious level didn’t want to take it out on the boy. This manifested itself as a change in muttering. I could see that McEnroe was still angry, but he was trying to accommodate the boy’s request for an autograph at the same time. Presumably a Nurturing Parent frame of mind was fighting for his unconscious attention, because the person in front of him was a child.
Unfortunately, with the boy and man both still in their excited (Child) state from the entire scenario – the exciting match, the crowd, and the presence of a tennis star – they were both moving into Adapted Child, keen to accommodate the star’s wishes but unable to decode what those wishes were. The result was that McEnroe attempted to indicate through his muttering that the boy follow him down the corridor and get his autograph, but that the adult go as far away as humanly possible! Because this was muttered, McEnroe still being angry at his opponent and himself, it was actually a very subtle piece of communication and only served to confuse both autograph hunters. Consequently, both followed the tennis player, who then turned on the adult and yelled at him in angry exasperation; his irritation at being pursued against his wishes became apparent.
Throw yourself into the following exercise and you’ll see what I mean. Imagine that you are sitting at a desk, working, when someone comes into the room and carelessly knocks over a drink that’s next to your work, ruining what you’ve done. How would you feel? How would you react? What would you think?
Depending on how vividly you created this scenario in your mind, you may be able to summon up a real feeling of loss, or anger, or frustration: some people report that they get a little sense of that “sick in the stomach” feeling. But the only real answer is: “It depends.” There are too many variables to know what frame of mind you would be in, and you will have made a number of assumptions or ignored a number of these variables in order to arrive at your “feeling” response. Your actual frame of mind at the time it happened would determine your actual response.
Scenario One: You have a meeting with your boss in five minutes where you hope to secure a promotion, and it is your boss who has knocked the drink over the work.
Scenario Two: You have just solved the equations of quantum chromo-dynamics (don’t ask me, I have no idea, it was mentioned on Wikipedia) and your neighbor’s child, who you didn’t even know was in the house and don’t like, has chased a bouncy ball into the room and knocked over the beverage.
I’m guessing that your response to the two scenarios would be dramatically different. The fact is that people respond very differently depending on the situation, their relationship with the other person present (subservient or dominant), and their prevailing mood at the time.
As you will see later when I explain the AFECT criteria I recommend for evaluating the real confidence you should have in any consumer insight, knowing that the right frame of mind has been involved when that insight is obtained is just as important as asking someone who has experienced whatever you’re interested in understanding. Not only does most research ignore the variation in frame of mind and its impact on how people fundamentally think, feel, or behave in a particular situation, it usually creates a “transaction” to stimulate a frame of mind that suits its purposes with total disregard for the frame of mind in operation at the consumer moment (or moments) concerned. Market research wants an answer, and in its effort to get one creates a new, unrepresentative mindset from which the respondent replies.
One common frame of mind elicited by consumer research44 is a balanced Adult ego state.
Researcher: “Please can I ask you some questions?”
Respondent: “OK.”
Researcher: “Which of these brands of ice cream do you buy regularly?”
This is very considered and very rational. It seems entirely balanced, fair, and even objective. It reflects the mindset not only of the person deconstructing the marketing issue to devise the questions the company wants answering, but also of the people who will hear the answers and consider what the company should do as a result. But what if you sometimes buy ice cream to eat because you feel sad, because your Child ego state needs cheering up? Eric Berne observed three distinct packages of thoughts, feelings, and behaviors. Which “package” is the respondent asked about ice cream going to call to mind? The process of research has done its best to ensure that they aren’t thinking or feeling in a Child frame of mind and the process of artificial post-rationalization has been further encouraged. Asking someone in a different frame of mind is like asking a different person.
Focus groups tend to have more variation in their encouragement of respondent ego state. Sometimes they invite the balanced (Adult) consideration that emanates from specifically inviting a group of people to a place and paying them to think about something and talk about it.
There is a very effective technique that Thomas Harris describes called “Parent Shrinking,” which can be used to shift people who are being belligerent and inflexible about a particular issue to a more balanced frame of mind; in other words, that will move them from Parent to Adult.45 Essentially, all it involves is remaining calm and balanced oneself and asking genuine questions. For example:
Person one (angrily): “I can’t believe you’d be so stupid as leave your coat at home today!”
Person two (calmly): “Why are you concerned about me and my coat?”
Person one (still angrily): “You’re going to catch your death of cold, you idiot.”
Person two (still calmly): “Are you worried that I might get ill?”
Person one (calming down): “Errr, yes, I don’t want you to be ill, it will spoil our holiday next week if you are.”
Person two: “I feel OK, actually, but perhaps we shouldn’t stay out too long.”
Person one: “OK, and let’s walk more quickly so you stay warm.”
This may sound a little contrived. In practice, it is actually very difficult to “break” the transaction that another person initiates46 because, before you’ve consciously shifted your position, you’ve been unconsciously primed either to be belligerent back (a Parent response such as “You’re not my mother, I’m perfectly capable of deciding what to wear”) or to respond deferentially (a Child response such as a sulky tut followed by an unconvincing “Sorry”).
If you consider an exchange during a consumer research depth interview or focus group where a respondent becomes angry about a brand or experience, you’ll appreciate the typical nature of this type of transaction. Research moderators and interviewers are trained to be dispassionate and balanced; after all, they are interested in understanding what the person thinks, and are not taking it as a personal attack. How do they react? They ask balanced questions, ostensibly to understand why the respondent feels so angry. The psychological consequence of this is that the respondent’s anger dissipates and they may well start opining a more balanced and reasonable position than the one they would otherwise hold and that they would “naturally” access in a real-life experience with the brand concerned. Which point in this transition will the researcher report?
Derren Brown recounts a more extreme form of this technique in his book Tricks of the Mind. When affronted by an aggressive and provocative drunk man late one evening, Brown describes derailing the potential attack using confusion. He said something to the man, in a calm and balanced (Adult) tone, that was completely unrelated to the aggressor’s line of thinking. The mental change of direction was sufficient to derail the train of thought of the person looking for a fight and shift the balance of power. Brown survived unscathed, bar the pain of a drunken chap recounting his unhappy night out.
I have seen consumer research group moderators deploy the same technique when they “move on” with an unrelated question from their list to stem the tide of a respondent’s critical outburst. In gaining control of the respondent’s frame of mind they may move further away from their true mental frame of reference. Ultimately, a reasoned response, while most pleasant to hear and potentially attractive to the audience for the research, isn’t necessarily accurate.
In the case of the Millennium Dome, an inherently rational interview process, comprising a number of questions, was evaluating what most people would approach as a “playful” day out. In posing the rational suggestion “Are you going to commemorate this historic moment by attending the main event this country is laying on to celebrate it?”, it is not surprising that so many people thought they might go along. However, had the organizers attached more weight to the number of people who actually make the effort to travel any distance to attend a themed event (such as a theme park or historic building) in any given year, they might have recognized that they were going to need to create something monumentally thrilling to cause six times that number of people to change their behavior and embrace the Dome as a new alternative.47
It’s essential to consider what frame of mind a person is likely to be in when they are really engaged in whatever the research has raised with them, and if they would even be mindful of it at all. If the frame of mind induced by the research is different, the response probably will be too.
The way in which people think about something is a by-product of the experiences they’ve had up to that point and how easy or difficult it is to think about. Unfortunately, it’s often extremely difficult to know which experiences someone will mentally associate with a question and how easy or hard they are finding it. However, depending on what associations they make, their responses can differ dramatically.
This was highlighted by a study that asked people to assess their risk of heart disease in the context of behaviors that either increased or decreased their likelihood of getting the condition. When participants without any history of heart disease in their family were asked to think of eight risk-increasing behaviors, the difficulty they had thinking of so many caused them to rate their own vulnerability lower than when they were asked to think of just three, and lower than when they had been asked to think of three or eight risk-lowering behaviors.48 When those asked were people with a family history of heart disease, who could be expected to think about the question more deeply as a result, the results were completely different. They perceived their risk as being higher when they thought about eight risk-increasing behaviours or three risk-decreasing behaviors.
Other studies have found that changing how a statement is presented, in terms of how easy or difficult it is to read on the page, influences the extent to which people believe it or not. Where people struggle with a question, or can’t be bothered to think about it, they will answer differently from when the question or the answer is easier to access.
In the research conducted for the Millennium Dome, potential visitors were classified as “persuadables” if they agreed with statements such as “I will decide nearer the time (if I’m going to visit the Dome).” All the potential mental complexity of deciding about a group trip to Greenwich and paying to visit a somewhat esoterically defined attraction could be neatly deferred by agreeing to “think about it.”
Frequently, a researcher’s training to “facilitate” a discussion will covertly encourage a particular type of response. If you have ever viewed a focus group, you may have noticed that very often the moderator will lean forward, sitting with very “open” body language, and look up slightly at the people seated in the horseshoe-arranged chairs. The moderator’s voice will be quite bright and bouncy. This is an understandable package of behaviors for the moderator to adopt: the purpose is to initiate a discussion on a particular subject and the open posture and bouncy tone say “I’m receptive; this is nice; talk to me.”
Unfortunately, this package of behavior has been observed to encourage a particular response. Work by clinical psychologist Kahler identified a number of packages of behaviors as a tool for assessing personality.49 One of them includes the following traits:
“Bouncy” high… but low… speech. Using expressions like “OK?,”
All right,” and “Hmmm?.”
A high tone of voice that rises at the end of a sentence.
Frequent head nodding, with hands reaching out and palms up
Leaning forward toward the other person (or people) and nodding the head.
Looks up under raised eyebrows, exaggerated smiles with teeth bared.
The list, which in my experience could just as easily be a training manual for focus group moderators, is classified as the “Please You” behavior driver. As the name suggests, the “return” on adopting this pattern of behavior is a greater chance of the recipient(s) liking you. This is understandable from the perspective of getting a group of strangers to start talking, but highly dubious if the intention is to have them be totally authentic. The problem for research is that using this driver has been shown to invite a similar response: the respondent likes the researcher and is inclined to say things that they believe will please them.
At a fundamental level, the moderator is saying: “Look how nice, unthreatening, and approachable I am.” The response it typically elicits is: “I’ll be nice too.” This is hardly a recipe for discovering fundamental consumer truths. I can only speculate what proportion of new product concepts have flown with flying colors through an evaluation process in focus groups because the moderator made everyone a little too at ease with him or her.
Another frame-of-mind distortion practiced in research comes from so-called projective techniques. For the uninitiated, this term covers a range of questioning techniques originally developed by psychologists that are designed to encourage respondents to go past the obvious and/or limited ways in which they might describe something, a brand for instance, and talk about it indirectly. Because respondents are presented with an ambiguous stimulus to respond to, their choice of response is presumed to reveal something of their underlying thoughts about it.
While opinions about the efficacy of projective techniques vary among psychologists, the main problem with their use in consumer research is the frame of mind they invoke in respondents. Consider the following request:
“I’d like you to pretend that this brand is a person. What would the person be like? What might they wear? What sort of car would they drive? Where would they live?” (I won’t go on, but you get the idea.)
People will typically react in one of two ways: either they will glaze over and not “get” the concept at all, or they will play the game and start providing answers. However, what they’re doing in transactional analysis terms is going into a Child ego state; they get over the fact that they feel silly and get into what’s being asked of them. This raises the question of whether their usual interaction with the brand is from a Child ego state. If it is, then the thoughts, feelings, and behaviors they tap into in connection with the brand and the projective exercise may be an accurate reflection of those associated with moments of consumption. If not, then however well intentioned, the answers they provide are probably not an indication of their real thoughts and feelings. Not for the first time, I would argue, the convenience of a technique that can be very good at providing research data – very good in the sense of the volume produced and how interesting it seems – has little to offer by way of dependable accuracy.
Creative questioning styles can provide more interesting responses, but they are not necessarily more reliable. If any of the research techniques used has induced a frame of mind that is not present during the actual consumer experience, it is unlikely to have obtained an accurate picture of what people think.
It might seem sensible for organizations seeking to understand what they’re doing well or what they should do next to ask their current customers to evaluate their products or services. When the results come back, who could blame them for supposing that such ratings provide a representative indication of how they are perceived or what people want? However, even once all the potential distortions of the research process discussed to this point have been taken into consideration, another issue exists: people who have gone to the trouble of purchasing something tend to value it more highly than people who haven’t.
This was one of the reasons I was able to predict accurately that the advice I gave some friends on what new car to buy would be ignored. They called to ask my opinion, as they were considering replacing their large car for something more practical and took my interest in the motoring section of their Sunday paper as an indication of expertise. I listened to their needs, did some research, and told them which model would, in my opinion, be best. Unfortunately, another couple who were friends of theirs had recently purchased a car different from the one I was advocating and had enthused to them about it. I suspected, rightly as it turned out, that my balanced appraisal would stand little chance against the post-purchase-endowed priming from their friends.
This phenomenon, known as the “endowment effect,” was first identified by Richard Thaler in 1980. As the experiment with the coffee mug in the previous chapter revealed, it only takes a few moments of ownership for people to value something significantly more highly. Another study, conducted in the late 1990s, highlighted just how powerful this effect can be. Obtaining tickets to a popular basketball game at Duke University was a major challenge: people had to overcome queues and win in a raffle to secure a ticket. Researchers asked people who had managed to buy a ticket what they would be prepared to sell it for, and compared this with the amount people who hadn’t managed to get one would be willing to pay. The difference was dramatic: people who had won a ticket valued it, on average, at $2,411; those who hadn’t won valued it at just $166.50 When it comes to ascertaining how much you can sell something for, the only reliable test is to try to sell it at a particular price and see what happens.
In its report on the Millennium Dome, the UK National Audit Office was quick to report the good news: the second point in its executive summary was that “87% of visitors were satisfied with their visit.”51 Leaving aside the somewhat abstract nature of the word “satisfied,” how should one interpret this statistic? Does it mean that the Dome experience was actually good and that the marketing agencies are to blame for the poor number of visitors? Does it imply that the critics were wrong in suggesting that it wasn’t very good? Or is it simply a reflection of the fact that, once they’ve paid for and experienced something, people may rate it far more highly than others who haven’t?
Any one of the 13 issues I’ve described can cause misleading research results, but when they work in combination with each other the impact can be extraordinary. Typically, the process of asking consumers what they think is either a one-off exercise or else repeated in a consistent fashion, resulting in no point of comparison, merely a consistent error that goes unnoticed. However, opinion polls are occasionally conducted with genuine independence, allowing us to see the potential range of outcomes created by compounding some of these question issues.
On September 10, 2009, an article entitled “Cut the TV Licence Fee by £5.50, Says the BBC Chairman” was published in the Daily Telegraph.52 The background to this issue is that a small proportion of the money collected from the £142.50 license for watching television has been used to help elderly and disabled people make the move over to digital television; once this switch has been completed the government is planning to use this money to fund local news on commercial channels. The article explained that an opinion poll conducted for the BBC Trust had found that “only 6 per cent of more than 2000 people surveyed for the Trust by Ipsos MORI supported the idea of using the surplus to help other broadcasters.”
This would appear to be a convincing argument for the government reconsidering its intentions, until you look at the questions they asked more closely and discover that in the course of the interviews three clear errors were made:53
Respondents were primed with information they wouldn’t normally be cognizant of regarding the level of the license fee and the fact that a proportion was being used to subsidize digital switchover.
The questionnaire forced certainty by not explicitly allowing people to say that they weren’t sure about the issue.
Of the six alternative uses for the money offered for consideration, only one was concrete (reducing the level of the license fee by £5.50), the remaining five were abstract (“Helping to increase…”, “Funding…” and “Spending more…”). Who could say what difference these might make to anyone’s life?
It’s also worth observing that while it is true that from the questionnaire used only 6% of respondents selected the option proposed by the government – funding other news programs – less than half of those interviewed said that they wanted to see the license fee reduced and 51% of people selected one of the five abstract options as their preference.
One week later, the Daily Telegraph published another article on the subject under the headline: “Most voters want BBC to share the licence fee.”54 This time it was reporting the UK government’s own 2,000-person survey of opinions conducted by TNS-BMRB on what people thought should happen to the license fee money that was becoming available. This time the polling company did at least check whether or not it was priming respondents with new information: apparently 71% of them weren’t aware that they had been funding the switch to digital television for the elderly and disabled. However, an appreciation that, through the interview process, the company was artificially informing the respondents didn’t stop it carrying on with the questions and publishing its results.55
According to the newspaper article, “Two thirds of those surveyed… said a proportion of the licence fee should be used to support regional news on other channels such as ITV.” One reason for the dramatic difference in results is that respondents weren’t given the option of having the money deducted from the license fee.
When the question was asked initially, 48% thought that the money should be used to support channels other than the BBC. Later, after a series of questions asking about the frequency they “watch/listen to/look at” national and regional news, the importance of having news provided by more than one source, and a statement explaining that the primary commercial channel, ITV, had said it may “no longer be able to afford to provide regional or local news,” a proportion of two thirds was achieved. Again, while the polling company did ask if respondents were aware of ITV’s statement about withdrawing its local news coverage, and learned that three quarters were not and had therefore been artificially primed by the poll itself, it didn’t prevent the government from publishing the results and, presumably, using them to inform its own decision making. Nor did it stop the polling company’s author stating in the report’s introduction that the questionnaire had been tested through a “cognitive pilot” so that “the questions would deliver an accurate and unbiased measure of public opinion.”56
So which poll is a correct gauge of public opinion? Neither. Rather, they both demonstrate perfectly how the process of inviting responses in research produces answers that are a by-product of the questioning process. Even if a standardized approach to such research could be agreed on, it would still be inherently flawed. The research process creates a focus that ordinarily doesn’t exist and wraps a frame around it that can’t help but shape the outcome.
The desire to solicit consumer opinion through questions is evidently a compelling one. Examples abound of contradictory poll results, inaccurate election forecasts, and feedback that appears to bear no relation to corresponding sales data. And yet the presentation of customer data as support or evidence in political argument and corporate decision making is perpetuated. The problem is a result of the human mind’s disposition for confirmation bias. I would go so far as to suggest that virtually everyone who regularly uses consumer research data has, on occasion, opted to ignore it or else dismissed it as wrong. As with any belief or superstition, people have no problem selectively discounting those occasions when they’ve become temporarily “research agnostic” and continue to behave in keeping with the belief that gives them comfort.
Our brains are highly adept at spotting patterns of cause and effect, but that means that we frequently misattribute chance events as having some underlying reason.57 When the data supports a decision that turns out well, that’s perceived as evidence that asking consumers questions is inherently worthwhile; when the findings from research are found wanting, they are soon forgotten. We need to acknowledge that, most of the time, we expect far too much from people in anticipating that they will be able to account for themselves and their opinions through a question-and-answer process that influences their thinking in such a way that it stops them from being the consumer we seek to understand.
These dynamic aspects of personality are largely ignored by consumer research. It prefers instead to subscribe to a constant or “average” theory of personality: people will do more or less what they do wherever they are and whatever else is going on. In fact, what people do and how they do it is not a given. They operate on the basis of cause-and-effect contingencies that are dependent both on the prevailing events at the time and on how the events are unconsciously processed and consciously interpreted.58
You might very well ask what one can reasonably ask and how one should ask it. This is a question that, commissioning influences aside, polling companies have been battling with since the 1930s. It is entirely reasonable to ask whether an answer, in terms of finding a valid opinion polling approach, exists at all. Is the constant fascination with soliciting opinions simply a byproduct of our own conscious delusions? We would like to believe that we know what we think and therefore it seems reasonable that others will know their own minds.
So are there any questions worth asking? And if so, when and how should we ask them?