17 Conclusions

This book has been an exploration of human strengths and capabilities. Despite the vagaries of experience, we can make reasonable sense of our worlds. Even when we do not achieve high levels of expertise, even when we confront uncertainties and other stressors, we generally find ways to reach and refine our goals.

I have concentrated on the strengths, the sources of power, that are difficult to study and understand. They are the abilities that repeatedly emerge in the study of naturalistic decision making. They are difficult to examine, as we have found, but that should not be grounds for ignoring them.

The sources of power we have been examining are what people do when they are not using deductive logic or probability theory. Work in naturalistic decision making over the past ten to fifteen years has attempted to give these sources a positive definition. We have tried to understand what people were doing and why their strategies might make sense, instead of seeing the lack of rationality as a failure of intellect.

Our efforts were aided by the fact that we were studying highly proficient decision makers. We came to respect and admire them. This admiration may have biased our work, or it may have informed it. When we study naive subjects who are performing unfamiliar tasks, and we know what the right answers are, then the best our subjects can do is not get it wrong. They are not going to surprise us, and it is unlikely that they will impress us. We can use our subjects to learn about our theories, but we will not learn very much from them. The relationship between experimenter and subject is between the one who knows and the one who is ignorant.

In our case, the relationship was reversed. The decision makers we studied were the ones who knew, and we were the ignorant ones, trying to find out. My colleagues and I could not call the people in our studies subjects. We used the terms participants or subject matter experts. Our motivation was not to test hypotheses, but to follow the lead of curiosity and get insights about the strategies the subject matter experts used.

Do not be misled by the types of experts you have met in this book: firefighters, jet pilots, nurses, and the rest. Each of us has expertise we use to make decisions, in one context or another. How do we decide which line will move faster at the grocery store? Which car will be best to follow through a traffic light? These are trivial examples. Countless others can be found at work, at home, or in school settings.

The sources of power discussed in the previous chapters are going against the general stream of research that emanates from the field of cognitive psychology. As long as the rational and analytical processes were defined as the ideal for reasoning, researchers have filled in more and more parts of the puzzle about how information is received and processed, and stored and retrieved. The computer is taken as a metaphor for the mind so that the challenge of the artificial intelligence framework is to find the right programs or the right architecture. Rational choice strategies offer the same type of general framework, treating decision making as the activity of judging probabilities and utilities. The job of researchers is to understand why people might generate the wrong values or inconsistent values. In both cases, artificial intelligence and rational choice strategies, the research agendas are well established.

In contrast, the processes we have been examining do not fit well into the frameworks of artificial intelligence or rational choice strategies. Rather than helping to tie off loose ends, the sources of power I describe in this book are busily proliferating them. We are finding that it was too early to reach closure about how people make decisions in natural settings. We had not properly understood the phenomena we were trying to model. We need to spend more energy in appreciating how people size up situations, make decisions, and solve problems.

Compiling the Assertions

Experience counts. This sounds so obvious that we should not have to waste time stating it. Yet most studies in decision making use subjects who are inexperienced in the task they are performing, and most advice assumes an inexperienced audience. The different sources of power covered in this book are ways of drawing on experience.

Expertise depends on perceptual skills. You rarely get someone to jump a skill level by teaching more facts and rules. Perhaps in a field such as mathematics, a teacher can move a student rapidly through some concepts. However, in natural settings, perceptual learning takes many cases to develop. Therefore, we cannot expect to grow instant experts by using powerful training methods. We can make training more efficient but cannot radically replace the accumulation of experiences.

The computer metaphor of thinking is incomplete. Mechanistic descriptions of skilled problem solving and decision making emphasize the storage, retrieval, and manipulation of data elements. This is one aspect of expertise, and certainly it is relevant to some tasks. But there are other aspects that are important.

Skilled problem solvers and decision makers are themselves scientists and experimenters. They are actively searching for and using stories and analogues, personal as well as borrowed from others, to learn about the important causal factors in their lives.

Skilled problem solvers and decision makers are chameleons. They can simulate all types of events and processes in their heads. They simulate the thinking of other people with whom they come in contact.

The sources of power described in this book operate in ways that are not analytical.

The sources of power described in this book have limitations as well as strengths. There are additional sources of power, such as analysis and calculation, that break tasks down into abstract elements and perform operations on these elements. In many difficult tasks, we blend the different sources of power and integrate them to fit the needs of the situation. I hope that the crude distinction between analytical and nonanalytical will give way so that we can learn to make more interesting comparisons and connections between the different sources of power.

Connecting the Sources of Power

In the preceding chapters we have examined several different sources of power:

We have also encountered a range of other judgments and abilities that could themselves have become the topics of chapters:

11307_017_fig_001.jpg

Figure 17.1 Sources of power

There are many different ways to connect all of these sources of power. Figure 17.1 presents one framework. Here, the two primary sources of power are pattern recognition (the power of intuition) and mental simulation. That is why they are so prominent. Storytelling seems to rely on the same processes as mental simulation, so it is tucked in next to it. The use of metaphors and analogues seems to rely on the same processes as pattern recognition, except that in pattern recognition the specific metaphors and analogues have become merged, so these two are pushed together. These are the four sources of power that refer to processes, to ways of thinking.

The other three sources of power are based on the first four, so they are arranged further into the periphery. These three refer to activities—ways of using the four processes at the base. Expertise (the power to see the invisible) derives from both pattern recognition and mental simulation. The ability to improvise in solving problems also derives from pattern recognition and mental simulation. Our ability to read minds depends on how well we can mentally simulate the thinking of the person. The additional sources of power are arranged according to the same sense of family resemblance. Figure 17.1 is a concept map of the way I am viewing these processes.

Is This Science?

The basic question of the book is, How do people make decisions and solve problems under natural conditions? The book describes the studies and efforts made to provide some answers. We can distinguish between a few different kinds of inquiry. A philosophical inquiry uses rules of logic to draw conclusions. A scientific inquiry uses carefully controlled and repeatable investigations. A pseudo-scientific inquiry pretends to conduct rigorous studies but does not, and it produces findings that are unreliable. What kind of an inquiry has this been? Clearly, it is not philosophical. In some ways it seems scientific, but the studies have weaknesses. That is one reason I spent so much time describing how we ran the studies, to allow you to judge for yourself how much confidence to have in the findings.

The studies I have described are not classical science. We did not calibrate our instruments and present stimuli that were carefully measured to determine their exact visual angle when projected onto the retina. These are the trappings of science but not the central features.

What are the criteria for doing a scientific piece of research? Simply, that the data are collected so that others can repeat the study and that the inquiry depends on evidence and data rather than argument. For work such as ours, replication means that others could collect data the way we have and could also analyze and code the results as we have done. It will be difficult for others to conduct interviews using our methods, but we have published articles describing our methods, so others can learn them. After all, I could not replicate experiments in gene splicing without a considerable amount of training, so the fact that training is needed in data collection to study naturalistic decision making does not present a logical problem. In recent years there have been studies replicating our findings, particularly the findings about the RPD model (Mosier, 1991; Pascual & Henderson, 1997; Randel, Pugh, & Reed, 1996).

Regarding the nature of our data, one weakness of our work is that most of the studies relied on interviews rather than formal experiments to vary one thing at a time and see its effect. There are sciences that do not manipulate variables, such as geology or astronomy or anthropology. Naturalistic decision making research may be closer to anthropology than psychology. Sometimes we observe decision makers in action, but we rely on introspection in nearly all our studies. We ask people to describe what they are thinking, and we analyze their responses. We do not know if the things they are telling us are true, or maybe just some ideas they are making up. We can repeat the studies or, better yet, other investigators can repeat the studies to see if they get the same results. Nevertheless, no one can confidently believe what the decision makers say.

The use of introspection raises questions about how much to trust the findings of studies. However, alternate methods of scientific inquiry have their own problems and limitations. Research on naturalistic decision making collects and reports data, and it can be used as a source of ideas and hypotheses. The think-aloud data are soft, and fuzzy, and they are difficult to interpret. Nevertheless, we can still learn a lot by observing and questioning people as they perform realistic tasks within natural contexts.

The rigorous nature of laboratory research increases our confidence that we can replicate the results, but the rigor does not ensure that we can generalize the results. Orasanu and Connolly (1993) have questioned whether findings that are carefully obtained under laboratory conditions apply outside the laboratory. They cited one study of songbird behavior, showing that the patterns of birdsong found in natural settings varied during the breeding cycle in reaction to courting, nest building, mating, and tending the young. Previously the species had been studied only in the laboratory, and the results had been inconsistent and uninterpretable. A second study they cited found that judges and parole officers made different decisions about criminal sentences when they were in the courtroom than when they were in a laboratory simulation.

Both the laboratory methods and the field studies have to contend with shortcomings in their research programs. People who study naturalistic decision making must worry about their inability to control many of the conditions in their research. People who use well-controlled laboratory paradigms must worry about whether their findings generalize outside the laboratory.

One way to evaluate a naturalistic decision-making approach is by its products: the nature of the ideas and models that it generates and the applications it produces. If NDM and the study of different sources of power turn out to make little difference, then we will lose confidence in the approach more surely than any debate over what is science.

Final Thoughts and Final Perceptions

Have you ever noticed, after you have spent an afternoon at an art museum staring hard at paintings and sculpture, that when you walk out, the world looks different from when you entered? Colors become brighter; contrasting shapes are more striking. We go to museums to see objects, but the process of seeing is affected, and that is one of the things we carry away when we are finished walking through the galleries.

One of my goals for this book was to change the way you see the events around you, even if just for a short time. The book has presented many arguments. But arguments can be refuted. I want to have an impact on your perceptions.

When you are with other people, listening to their stories and telling your own stories, you may find yourself listening and speaking differently.

When you give someone instructions or receive instructions for carrying out a task, you may begin appraising whether any important aspects of communicating intent have been left out, or you may appreciate the skillful way some characteristics of intent were described.

When you imagine how something could have occurred or imagine how to make something happen, you may be more aware of how you are building and playing with mental simulations.

When someone uses an example, in an argument or to solve a problem, you may think of it as an informal experiment, an analogue, a carrier of experience.

When you have a chance to work with an expert, you may watch more closely to find out what that person is seeing that you cannot.

When you are a member of a team, you may notice the way the diverse experiences are voiced and blended. You may find yourself watching the way thoughts are created and combined, appreciating the chance to reflect on your own mind and on the sources of power you draw on with so little effort.