2 The Big Puzzle

Intelligent behavior in people is a product of the mind. But the mind itself, of course is not a thing; it is more like what the human brain does. The actual physical object that lies behind our external behavior is the brain.

The problem

Although we know much more about the human brain than we did even ten years ago, the thinking it engages in remains pretty much a total mystery. It is like a big jigsaw puzzle where we can see many of the pieces, but cannot yet put them together. There is so much about us that we do not understand at all. Just how are we different from the rest of nature? What makes us so darn smart?

People in different areas of research come to this puzzle from different angles, each holding some of the pieces. Not too surprisingly perhaps, each of them argues that their perspective is the important one, the one that really matters. Here is the sort of thing you might expect to hear:

• What we need to care about is language, since the presence of a mind is most clearly manifest in the ideas it expresses. We know a mind by listening to what it says, reading what it writes. We are unique in the animal kingdom in recording what we know, what we have learned, how we think. We will never get to study the brain of Shakespeare, but we get to experience the mind of Shakespeare every time we tackle one of his plays.
• What we need to care about is psychology, how a mind works in people and how that compares and contrasts with other animals. People’s intuitions about how they think is quite different from how they really do think when studied closely in a controlled setting. Memory, learning, perception, attention, cognition, affect, consciousness, these are all psychological categories and together make up the mental life of people.
• What we need to care about is neuroscience, how a brain organizes itself to make a mind. When we talk about the mind, what we are really talking about are the states and transitions of a large assembly of highly connected electrochemical com­ponents. As we learn more and more about the brain—and the recent progress here has been phenomenal—there will be a revolution in how we talk about the mind, just as there has been a revolution in how we talk about life since the discovery of DNA.
• What we need to care about is evolution, how the human species developed under evolutionary pressure. Nothing in the human species, including its brain, stands alone. Every human faculty is the product of evolution, and correlates can always be found in other species. The human mind is a Swiss army knife of adaptations, and one cannot understand it without understanding how it evolved and how it connects to reproductive success.

And there are others.

I believe a certain amount of humility is called for when we talk about things like the brain, the mind, thinking, and intelligence. Listening to scientists brimming with confidence in full promotion mode, it may appear that new ideas, new approaches, new techniques, new tools are putting us on the threshold of an era where we will crack the mystery. I think this is quite wrong. The mistake, in fact, is to think that there is a single mystery to crack. What we have is a collection of mysteries, a jigsaw puzzle of pieces that need to be brought together and assembled from many different perspectives before we can truly appreciate the overall picture.

We need to train ourselves to be skeptical of any research group that insists that one part of this puzzle is the true core, the key to the whole thing. We should be even more skeptical of anyone who claims something like “We are on the verge of figuring out how the brain works.” This should sound to us like someone claiming to have figured out how weather works, or how the stock market works.

Just for reference, let us call the issue of confusing a handful of pieces for the entire puzzle the Big Puzzle issue. (It will come up again often.)

Given this Big Puzzle issue, how are we then to talk about thinking at all without falling into pompous oversimplifications? I believe the answer is to accept that we are dealing with only one part of the puzzle, and to do what we can to see how the pieces we are holding fit together, while resisting the temptation to suggest that the rest of the puzzle is just more of the same. The mind is staggeringly complex, but this should not rule out looking at a small part of it in detail.

So what part of the puzzle will be discussed in this book? To jump the gun somewhat, this is the story that will be told here:

• What we need to care about is intelligent behavior, an agent making intelligent choices about what to do. These choices are made intelligent through the use of background information that is not present in the environment the agent is acting in. This background information is what we call knowledge. The application of knowledge to the behavior at hand is what we call thinking. The problem that needs to be sorted out is how, in concrete terms, this all works, how background knowledge can make a difference in an agent deciding what to do. The solution we consider is a computational one. In the same way that digital computers perform calculations on symbolic representations of numbers, we suggest that human brains perform calculations on symbolic representations of knowledge, and then use the result of those calculations to decide how to act.

It should be clear that this story is very far from proposing a solution to the entire puzzle. If it were, where would emotions like modesty, envy, and grief fit in? What about the social or interactional aspects of intelligence? Where is perception and imagination? Or daydreams and fantasies? What parts are uniquely human? What accounts for degrees of intelligence? And what about different kinds of intelligence (like emotional intelligence)? Or the effect of drugs and mental illness? And where is consciousness in all this? Or creativity? And what about spirit and pluck, the stuff that makes Molly Brown unsinkable?

These are all great questions that I will not be talking much about. Yet I do not want to give the impression that what is left is small or beside the point. Thinking, even in the very narrow sense described above, is still as broad as the ideas that can be thought about. And we can think about anything! Despite all the amazing scientific progress to date, we still have no clear sense of what happens when we think about who will win the Academy Award for Best Actor, or how thinking about Debussy is similar to but different from thinking about Ravel, or what it is like to change your mind over a period of time about whether a free market needs to be regulated.

The mystery remains even in the small fragment of the puzzle considered here.

The difficulty

Before leaving this topic, it might be worth looking in more detail at why areas like psychology or neuroscience by themselves are unlikely to be able to tell us what we want to know about how the mind works. To make the point, we will use a thought experiment involving a simple machine that will be much easier to analyze.

Imagine a small device connected to a beeper, a lightbulb, and a keyboard. Anytime someone types a two-digit number on the keyboard, the machine responds with some beeps and some flashes. Call the machine M. Think of M as a very simple brain, with the keyboard as its only sense organ, and the beeper and lightbulb as its only effectors. Imagine that your job is to figure out why it is beeping and flashing the way it does. The exercise is useful because it gives us a good feel for what it might be like to figure out what is behind intelligent behavior.

Let us say, for example, that the numbers 37, 42, 53, 16, and 37 are typed in sequence on the keyboard. M produces the behavior shown in the following table:

Input	Beeps	Flashes
37	1	6
42	1	8
53	2	8
16	2	2
37	3	2

Now given all this, what makes it behave the way it does?

I am going to start by giving away the answer. M is a small digital computer attached to the keyboard, beeper, and lightbulb. It repeatedly takes as input the number typed on the keyboard, and produces as output a number of beeps and of flashes according to a tiny computer program.

The entire program that controls M is displayed in figure 2.1. (Readers who do not wish to try to read this small computer program can also skip the rest of this paragraph.) To see how it works, note that it uses integer arithmetic, where mod means the remainder after division. So the value of 37 / 10 is taken here to be 3, and the value of 37 mod 10 is 7. For example, when the 37 is first typed on the keyboard, M produces 1 beep and 6 flashes. Here’s why: according to the program in figure 2.1, the W is set to 37, the X is set to 3, the Y is set to 3 × 3 + 7 = 16, where 16 / 10 = 1 and 16 mod 10 = 6. When 37 is typed on the keyboard the second time (the fifth input), the result is 3 beeps and 2 flashes because the U at that point is 5 (from the third input), and since 5 > 3, the Y is set to 5 × 5 + 7 = 32.

Figure 2.1 The code for a mystery machine

So that’s the secret. Now having seen it, pretend that you know nothing about the program in figure 2.1, but that your job is to understand why M does what it does.

We can imagine being a psychologist, running some experiments on M and observing its behavior. There are fewer than a hundred possible inputs here, but even in this very simple setting, our life is complicated by the fact that M has memory and makes decisions about how to behave based not just on the last number it saw.

To get an idea of what it is like for a real psychologist, we have to imagine that the number of tests we can run is much, much smaller than the range of possible inputs available to the machine. Think of a reading comprehension test (like the kind to be discussed in chapter 4), and compare the number of sentences in the test to the number of sentences that subjects will read in their entire lives. Psychology must live with evidence drawn from very small parts of the total behavior space.

For example, let us simply generalize M so that instead of taking as input a two-digit number, it takes as input a ten-digit number. Now instead of 10² (one hundred) possible inputs, we have 10¹⁰ (ten billion) of them. If we observe that, because of memory, we may also have to consider what the machine saw in the previous step and the one before that, then we need to consider 10³⁰ possible sequences. If the memory could go back further and depend on the ten most recent inputs, there are now 10¹⁰⁰ sequences to sample from, more than there are atoms in the known universe.

So it does not take much to rule out the possibility of observing a significant range of stimulus and response. The sensory environment of M, simple as it is, and its memory, simple as it is, overwhelm any sort of comprehensive testing.

This, in a nutshell, is what makes psychology so hard.

It is extremely difficult to design an experiment constrained enough to provide meaningful results. If my subjects are Jim and Jane, how am I going to control for the fact that they have had very different lives, seen very different things, and come to my experiment with very different beliefs and goals? Not too surprisingly, some of the most revealing psychological experiments involve perceptual tasks where an immediate response is required, in the milliseconds, say. This is quick enough that long-term memory, which will be quite different for Jim and for Jane, plays a less significant role. When a subject gets to sit back and muse for a few seconds, it is extremely difficult to control for all the variables.

In a way, psychology is handicapped by the fact that it gets to look at subjects only from the outside. External stimuli can be presented, and external responses can be observed, but that’s it. It is not ethical to open up a living person’s head and attach electrodes here and there to see what is happening. Some of our best understanding of brain function has come from cases where the brain was exposed as a result of an operation—severing the corpus callosum to control epilepsy, for example—and asking patients to describe what they sense when parts of the brain are stimulated.

However, there is new technology, such as fMRI, that is much less invasive than brain surgery and is giving us a much better picture of many parts of brain function. We can see that the part of the brain involved in motor control is activated even when a subject is just thinking about physical activities. We can see that the part of the brain that is active when a person is swearing is not the one that is normally active during polite language production. All these very impressive developments in neuroscience suggest that given sufficient time, we will understand how beliefs and goals come together in thinking to produce human behavior.

To see why we should nevertheless be skeptical about this, let us go back to M. Imagine a neuroscientist whose job it is to determine why M behaves the way it does. Like the psychologist, the neuroscientist is not told about the program in figure 2.1. Unlike the psychologist, however, the neuroscientist will be given access to the internal workings of M, as if it were a functioning brain.

When M is opened up and studied in the lab, the neuroscientist can see that it is an assembly of standard electrical components powered by a battery. As digits are typed, some of these components are activated and others stay dormant, perhaps only rarely lighting up. As more and more numbers are typed on the keyboard, some tantalizing patterns begin to emerge. The question is: do we expect the neuroscientist to crack the puzzle about M’s behavior?

It is certainly true that M is nothing more than a collection of electrical components. Any behavior it produces is ultimately due to those components being in one state or another. If M were a brain we would say that the state of the brain is what determines how we behave; anything else we might choose to talk about (beliefs, goals, emotions, a mind, whatever) must be realized in one way or another in the state of the brain.

But the question is whether we will be able to recover the regularities in M’s behavior in the properties of M’s components. For example, we might want to be able to discover things like the fact that M squares the first digit in the number it sees but not the second. Will we see this in the electrical components?

There is good reason to believe that we will not. Let us suppose that the neuroscientist is talented and lucky enough to extract the entire program that M is running by carefully studying the state of all the electrical components over a period of time.

Here’s the problem. M squares the first digit (or a previous first digit) because of lines 8 and 9 of the program in figure 2.1. But that program may not be located anywhere in M’s memory. It is typical of computers that programs are first translated into another form that is more easily executed by hardware. In computer jargon, the program in figure 2.1 is called the source code, and the translated version that is stored in the memory of M is called the object code. At the very best, the talented neuroscientist gets only the object code, and there is no reason to suppose that having the object code in hand would allow someone to recover the source code that generated it.

For example, squaring a number on a computer is typically not a single operation. Multiplication may show up as a large number of operations in object code. (There are clever forms of multiplication that perform much better than the digit-by-digit form we were taught in grade school.)

Even worse, the numbers themselves may not be encoded in a simple way in the states of the electrical components. Multiple components may be involved and they need not be physically close to one another. Indeed, in so called distributed representations (used in some neural net models of the brain) it may be necessary to look at the state of many electrical components to find the value of any one number represented. Worst of all, in a distributed representation, the state of a single component may be involved in the representation of more than one number.

So while translating from source code to object code is typically easy, going from object code back to source code is like trying to crack a big cryptographic puzzle. Companies that sell software products protect their intellectual property by releasing object code only, confident that this “reverse-engineering” is difficult enough that it cannot be solved in an economically feasible way. (Software products that are “open source” are counterexamples where the source code is made public as well.)

So even with the finest test equipment and electrical probes, the neuroscientist may not be able to recover M’s original source program. Having unrestricted access to all of the components in the lab, even when the components themselves are assumed to be well understood, is no guarantee that we can figure out something as simple as why M behaves the way it does.

This, in a nutshell, is what makes neuroscience so hard.

Even if we had total access to the hundred billion or so neurons making up a human brain, and even if we were able to treat those neurons as idealized, noise-free, digital components, we might still not be able to figure out why we behave the way we do. When a neuroscientist has to deal with real neurological components, not electrical ones, where there are a number of confounding chemical and biological processes going on, the situation is that much worse. How do we remember certain kinds of facts? How do we assemble those facts to draw new conclusions? How do we use these conclusions to decide how to behave? All of these are clearly tremendously more complex than figuring out why M is beeping three times. It is asking too much of neuroscience, even a wildly more advanced version of neuroscience, to tell us what we want to know.

We need to look elsewhere.

An approach

When faced with the problem of making sense of a complex phenomenon—and human thinking is certainly one of them—we actually have two options: we can study the objects that produce the phenomenon (that is, human brains), or we can attempt to study the phenomenon itself more directly.

Consider, for example, the study of flight (in the days before airplanes). One might want to understand how certain animals like birds and bats are able to fly. One might also want to try to build machines that are capable of flight. There are two ways to proceed:

• study flying animals like birds, looking very carefully at their wings, their feathers, their muscles, and then construct machines that emulate birds;
• study aerodynamics—how air flows above and below an airfoil, and how this provides lift—by using wind tunnels and varying the shapes of airfoils.

Both kinds of studies lead to insights, but of a different sort. The second strategy is perhaps the more general one: it seeks to discover the principles of flight that apply to anything, including birds.

Thinking is similar. Although we do want to understand human thinking and how it leads to intelligent behavior, this does not mean that there is no choice but to study humans. While there is a lot to be learned by studying the human brain (and other brains too), we can focus on the thinking process itself to determine general principles that will apply to brains and to anything else that needs to think.

This is what the philosopher Daniel Dennett calls taking a design stance. What we attempt to do is see what would be involved in designing something that is capable of flying or thinking. Instead of getting caught up in the details of how human brains do the job, we shift our attention to the job itself, and ask how it might get done at all. The hope is that in so doing, we get to see what is and is not essential in producing the phenomenon of interest.

Of course a design stance will not solve the entire puzzle. It may not tell us much about how the phenomenon can be the end result of an evolutionary process, for instance. And it is not applicable at all to phenomena that cannot be observed. If what we care about is not how birds are able to fly, but whether they have a certain internal sensation that they do not display, then a design stance will be of no help.

So in the end, what we will be concerned with here is observable intelligent behavior and how it is produced. We will argue that thinking is needed, of course, but we will not be concerned with what the result of this thinking might feel like to the agent doing the thinking.

For many researchers, it is the subjective feeling of conscious thought (sometimes called qualia) that is the truly interesting and distinctive feature of the human mind. I take this to be a Big Puzzle issue. The sensation of consciousness is an interesting part of the puzzle, no doubt, but it is not the only one.