FIGURE 32. A comparison of visual perception and mental imagery in the V1 visual map.
Paul Kim
THOUGH OUR VARIOUS SENSES and manifold movements are remarkably diverse, they all reflect concrete physical phenomena. If perception and action are products of the here and now, it would be reasonable to conclude that sensory and movement maps are limited to representing the moment-to-moment events happening around and within us. And yet they are not. Quite the opposite, in fact—brain maps can transport us beyond the here and now, representing things that happened long ago, two minutes ago, or may never happen at all.
The most dramatic example I’ve seen of the brain’s transportive power comes from the description of a young woman on what may have been the scariest day of her life. Wilder Penfield referred to her as MM, but I will call her Miriam. She was awake and alert while a surgical team prepared her for surgery. Her scalp and skull were opened up and her brain exposed, so that Penfield and his colleagues could search for the source of her debilitating seizures. Unlike Penfield’s patients whom we have already met, Miriam had seizures that did not begin with movements. Rather, she would be overcome by a “feeling—as though I had lived through this all before.” At other times, she would experience flashbacks to previous moments in her life, sometimes moments that she could no longer otherwise recall. After these sensations, she would walk about in confusion or speak unrelated strings of words—actions that she could not remember performing once the seizure was over.
In search of the origins of her seizures, Penfield probed the right side of her brain with an electrode. He began stimulating regions of the temporal lobe, inching inward toward the hippocampus, a structure that lies beneath the cortex and is profoundly important for memory. After one stimulation, she said, “I think I heard a mother calling her little boy somewhere. It seemed to be something that happened long ago.” Asked to explain what she heard, she added, “It was somebody in the neighborhood where I live.”
When Penfield stimulated the same place again, she said, “I hear the same familiar sounds. It seems to be a woman calling, the same lady. That was not in the neighborhood. It seemed to be at the lumberyard.” And then she added, “I’ve never been around the lumberyard much.”
Penfield pressed on, probing new spots in her brain with the electrode. When he stimulated another place, she said, “I hear voices. It is late at night, around the carnival somewhere—some sort of traveling circus. I just saw lots of big wagons that they use to haul animals in.”
At another site: “Oh, I had the same very, very familiar memory, in an office somewhere. I could see the desks. I was there and someone was calling to me, a man leaning on a desk with a pencil in his hand.”
Penfield began narrowing in on the site where the seizures were originating. Working slowly but confidently with his scalpel, he cut out a large chunk of cortex from the side of the brain and then tested the deeper tissue that now lay exposed. With one stimulation, she said, “I feel very close to an attack. I think I am going to have one—a familiar memory.” With another: “Oh, it hurts and that feeling of familiarity—a familiar memory—the place where I hang my coat up, where I go to work.”
That is the last we hear from Miriam in her own words. Penfield was finished using her responses and her evoked experiences to locate the damaged tissue where her seizures began. He found it close to the hippocampus. The tissue was hardened, perhaps compressed due to a complication that happened when Miriam was born. Although Penfield did not say, presumably his scalpel went deeper still to cut out the rest of this damaged tissue. We can only hope that Miriam went on to survive the surgery, recover, and return to a better life.
Aside from the dramatic circumstances of Miriam’s surgery and the notion that a piece of her brain was removed in the process, something else about Penfield’s account is unsettling. Miriam’s first-person recounting of her peripatetic mental experiences seems strange and almost magical. With a tiny jolt to her brain, she was transported to different times and places. Her senses informed her of things that weren’t there. Perched in the sterile operating room, surrounded by drapes, gauze, and medical staff, she heard the voice of someone’s mother, saw wagons and strange men, and felt that she was somewhere else. The whole thing sounds like science fiction.
But though Miriam’s circumstances were extraordinary, her experiences beyond the here and now were not. There are many ways in which we are transported, or willfully transport ourselves, to other places and times every day. In fact, we spend about half our waking hours thinking about something other than what we are doing or perceiving at the moment. We regularly hear, see, and feel things that aren’t before us. We use our imagination to conjure fictional events or possible futures. We call past sensations to mind, remembering the expression on someone’s face or the sound of their voice. We focus on recent sensations or actions, as when we mentally repeat a shopping list or mentally retrace our steps to find a lost set of keys. At night, we succumb to dreams filled with sensations and actions, creatures and emotions—as all the while we drool on a pillow in the dark. Why should we have the capacity to experience things that are not happening right now, at this very moment? And how do our brains carry out this impressive feat? As you might have guessed, the answer lies, at least in part, with the brain’s cornucopia of maps.
On its face, there would seem to be nothing in this world less amenable to scientific study than mental imagery. It is unobservable and immaterial, culled in an instant from nothing, only to blink just as quickly back out of existence. What is imagery? Who has it? And how does it work? These are difficult questions to answer. How do you study something that cannot be objectively seen, felt, or measured? And why would scientists be brazen or foolish enough to even try?
One of the first bold attempts to pin down mental imagery and subject it to scientific scrutiny took place in England in the 1870s. People were asked to fill out a questionnaire with unusual mental exercises like this one:
Think of some definite object—suppose it is your breakfast-table as you sat down to it this morning—and consider carefully the picture that rises before your mind’s eye.
Illumination.— Is the image dim or fairly clear? Is its brightness comparable to that of the actual scene?
Definition.— Are all the objects pretty well defined at the same time, or is the place of sharpest definition at any one moment more contracted than it is in a real scene?
Colouring.— Are the colours of the china, of the toast, bread crust, mustard, meat, parsley, or whatever may have been on the table, quite distinct and natural?
Although your morning meal may not have involved china or bread crust, presumably you can conjure a memory of where you were and what was around you earlier today. However you experience this memory—how you may see it in your mind’s eye—will seem normal to you. But does everyone remember and visualize past places and events in the same way? Before this quirky questionnaire was developed, no one had bothered to ask. Other exercises in the survey, like the one that follows, took on other forms of mental imagery to explore how its respondents experienced them through different senses.
Call up before your imagination the objects specified in the six following paragraphs, numbered A to F, and consider carefully whether your mental representation of them generally, is in each group very faint, faint, fair, good, or vivid and comparable to the actual sensation:—
A. Light and colour.— An evenly clouded sky (omitting all landscape), first bright, then gloomy. A thick surrounding haze, first white, then successively blue, yellow, green, and red.
B. Sound.—The beat of rain against the window panes, the crack of a whip, a church bell, the hum of bees, the whistle of a railway, the clinking of tea-spoons and saucers, the slam of a door.
C. Smells.— Tar, roses, an oil-lamp blown out, hay, violets, a fur coat, gas, tobacco.
D. Tastes.— Salt, sugar, lemon juice, raisins, chocolate, currant jelly.
E. Touch.— Velvet, silk, soap, gum, sand, dough, a crisp dead leaf, the prick of a pin.
F. Other sensations.— Heat, hunger, cold, thirst, fatigue, fever, drowsiness, a bad cold.
The entire questionnaire, from the mustard and parsley to the crisp dead leaves, was the brainchild of Francis Galton, an expansive thinker with his fingers in many pies. He studied and wrote about statistics, meteorology, human perception, and psychology. Today he may be best known as a founder of eugenics, a pseudoscientific and political movement that declared some races genetically and mentally superior to others and advocated societal interventions to promote “superior races” and their “superior genes” at the expense of everyone else. This horrifying movement was used to justify racial subjugation, forced sterilization, and even genocide.
Galton himself coined the term eugenics in one of his books. Later, in the very same book, Galton devoted an entire chapter to revealing the results of his groundbreaking survey on mental imagery. The questionnaire had been circulated to hundreds of people, including scholars, artists, schoolchildren, and passersby on the street, but Galton paid special attention to a sample of responses from a hundred adult men, “at least half of whom are distinguished in science or in other fields of intellectual work.” To Galton, this sample represented the best that humankind had to offer: aristocratic European men.
If Galton had hoped to find that these ideal subjects all experienced either strong or weak mental imagery, demonstrating which one was the hallmark of superior intellect, he was sorely disappointed. Their ability to conjure mental images ranged the full gamut. For example, one reported, “All the objects in my mental picture are as bright as the actual scene.” Another described imagery that was “fairly clear, but not equal to the scene.” A third replied, “My powers are zero. To my consciousness there is almost no association of memory with objective visual impressions. I recollect the breakfast-table, but do not see it.” The most apparent generalization to be made from Galton’s sample was that very little could be generalized. Individuals differed substantially in how vivid and clear their mental imagery was, if they were able to generate such a thing at all.
Galton’s examination of mental imagery was groundbreaking for its time, reporting the first study to offer an in-depth assessment of people’s abilities to generate imagery. It also described the pioneering use of a format that has now become a hallmark of modern research: the questionnaire. But like many of the early scientists in psychology and neuroscience, Galton sought to open windows onto the human mind and brain while closing the door on most of humanity. His work reminds us that the pursuit of scientific truth is always embedded within a context—a time, a place, and a tangle of interlocking beliefs and biases. The challenge is looking with a ruthless eye through what the past has left us, to sift and sort: to both recognize where there is value and to clearly see and call out where hatred and fear parade about under the mantle of science.
From Galton’s survey, and considerable work that has come since, we know that people do not all experience mental imagery in the same way and to the same degree. But most people do experience mental imagery in some form and to some degree. Some people even experience such vivid imagery that they find it distracting or disruptive. In all cases, something sparks these imagined sensations: it might be a wishful daydream, following Galton’s written exercises, or even receiving an electrical jolt directly to the brain, as was the case for Miriam. Imagery can be triggered in many ways. But what is it made of, and why does it feel so much like perception?
The answer to this question is surprisingly simple. Mental imagery feels like perception because it engages many of the areas of the brain involved in actual perception, and it engages them in much the same way. Modern technologies like functional MRI brain scans have led to breakthroughs in our understanding of how mental imagery works. These scans revealed that activity within the V1 visual map itself reflects the contents of a person’s mental imagery, thereby literally making a picture of the imagined content in the person’s brain. The illustration in Figure 32 shows how imagery and perception generate similar patterns of activity within the V1 visual map, although the activity is weaker for imagining than it is for actually seeing.
V1 is by no means the only visual area of the brain involved in making mental imagery. For example, V1’s neighbor, a visual map nicknamed V2, is also in on the game. Imagining visual motion also ramps up activity in a motion-preferring zone of the visual cortex. Imagining a face boosts activity in the FFA face zone, while visualizing a place activates the PPA place zone. Both within and across brain maps, imagining something involves weakly activating the same bits of brain required for you to see it.
Consider that for a moment. Vision is triggered from patterns of light entering your eye and stimulating your retina. Mental imagery is generated entirely from within, by a dark stew of brain cells that have never been touched by so much as a single ray of light. And yet that dark stew can spark nearly the same type of activity in your visual cortex as your eyeballs can. In V1, that activity forms a literal picture of what you imagined, although drawn with firing frequencies rather than ink. As far as your visual cortex is concerned, imagining resembles a weaker form of actual seeing.
And that is just the beginning. You can use your imagination to generate imagery for senses other than vision as well. Whatever type of imagery you generate, you do so by activating the appropriate brain maps, mimicking the specific activity that would arise for actually perceiving the imagined thing. When you imagine a sound or hear a song in your head, this imagery is evident in the activity within sound-frequency maps, including A1. When you imagine being touched on a part of your body, touch maps, including S1, allow you to feel that imagined sensation. When you imagine speaking, you are generating activity in areas that contribute to actual speech production. When you imagine moving your fingers or your hand, neurons in sectors of your motor cortex, including the hand region of M1, ramp up their firing rates.
FIGURE 32. A comparison of visual perception and mental imagery in the V1 visual map.
Paul Kim
This neural mimicry is extraordinary in its own right, but it is all the more remarkable when you consider how often it is happening in your brain. For most of us, mental imagery dominates the time we spend immersed in thoughts beyond the here and now. Daydreaming is one obvious way that mental imagery finds its way into our daily lives. Daydreams can help us consider and prepare for possible future outcomes. But imagery is not just the stuff of daydreams. For example, people often experience visual, auditory, or other kinds of imagery while reading or listening to narrative passages or stories. This imagery is accompanied by activity in brain maps for vision, hearing, and even touch. By way of this imagery, you can virtually see, feel, and hear the world that the storyteller or author has created for you.
In the same vein, when most people retrieve memories, they experience imagery of past people, places, and events. Activity in your visual, auditory, and other brain maps represents this recollected imagery just as it would for imagery manufactured out of whole cloth from your imagination. If we weren’t so accustomed to using mental imagery to access information about the past, we might appreciate how remarkable a feat this is. You may not be able to visit your childhood bedroom in person; perhaps it has even been razed to the ground. Nonetheless, most people have the ability to feel as if they are in that room again—to see its layout and perhaps notice details in the objects around them. As far as your mind and your brain are concerned, you are practically there, even if the place no longer exists outside your mind.
The immersive nature of memory can be wonderful or terrible, depending on whether the memories are ones that you care to revisit. For example, people suffering from posttraumatic stress disorder or depression often experience intrusive and upsetting imagery of past events. These painful or frightening recollections would not have such power if memory were a simple recounting of what happened and not how it happened—and crucially, how it felt to have it happen to you.
Mental imagery allows us, for better or worse, to reach back into our distant past. But we also harness it to keep track of things that have only just happened. Psychologists call this working memory—the ability to keep information “on hand” for a short period of time. Let’s say you see a speeding car crash into a parked car and then careen away. You are the only witness. You glimpse the license plate number: NJ612B5. You need to remember the number until you can write it down or report it. What do you do? If you don’t have a pen or a cell phone handy, you might repeat the string of numerals in your head: NJ612B5. NJ612B5. NJ612B5. This inner speech is a kind of mental imagery.
Psychologists have studied this phenomenon for the better part of a century and have made several intriguing observations. It is not the meaning nor the written appearance of the words or numbers you hope to remember that matters for this type of working memory, but rather how the words or numbers sound when spoken aloud. Trying to remember a string of words that sound similar, like bite blight trite try tine is much harder than remembering words in which each sound is different. Likewise, it’s the length of the word when spoken, rather than its length on the written page, that matters for this kind of working memory. People can only mentally rehearse content that takes less than about two seconds in total to say. If it runs longer than that, they forget the beginning by the time they reach the end.
Studies using functional MRI brain scans have revealed the role of brain maps in working memory and, in doing so, helped to explain its peculiarities. These studies showed that you use body maps in your motor cortex and sound-frequency maps in your auditory cortex to imagine speaking and hearing the content, respectively. That’s why actually talking or singing interferes with your ability to mentally rehearse; real talking and imagined talking use many of the same bits of brain. Likewise, listening to someone else actually speaking interferes with your ability to represent the sounds of your own inner speech using the auditory cortex. In short, these brain areas can process speech movements or sounds that are either real or imagined, but they fail dreadfully at doing both at once.
We see a similar process at work in a second form of working memory that you might use to maintain visual information, like the pattern on a fabric or the layout of pieces on a chessboard. Looking at an image, imagining that image, and maintaining that image in working memory all evoke activity in the same bits of brain within your visual maps. And just as working memory based on inner speech is disrupted when you hear actual speech, your ability to hold an image in mind is disrupted when you visually inspect a different image. The neural representations of the two different images interfere with each other in your visual maps.
Whereas recollection, working memory, story comprehension, and daydreaming transport us beyond the here and now while we are awake, a different type of imagery transports us while we sleep. Although dreams are not the same as the mental imagery you willfully conjure in the daytime, the two bear a family resemblance. As with imagination and recollection, dreams are triggered by signals from areas of the brain, such as the hippocampus, that use distributed codes to represent events, actions, places, and times. And in all of these cases, those signals act upon sensory and movement brain maps, driving their activity and, in turn, generating your experience of feeling and moving when you are actually doing no such thing. As with mental imagery, dreaming entails activity in brain maps that parallel the contents of the specific dream. For example, neurons within your FFA face zone ramp up their activity when you dream of seeing a face.
The fact that dreams and imagination play out in your brain maps reveals the far-reaching importance of these structures and their zones and boundaries. The very brainscapes that you rely upon to perceive your world also shape and distort the contents of your dreams, memories, and fantasies. This is why, for example, you cannot see out of the back of your head, even when you are asleep. Your visual brain maps devote no territory to representing the space behind your head, leaving no such space for dreams or imagination to occupy.
In fact, the idiosyncrasies of a person’s brain maps, such as the relative overall sizes of specific maps, appear to affect the precision of their mental imagery. Because neurons within a large V1 visual map tend to each have smaller receptive fields, people with relatively large V1 maps tend to have higher visual acuity, or the ability to see fine-grained detail. Remarkably, people with larger V1 maps also tend to have greater visual working-memory capacity and better precision for location information in their visual mental imagery.
The foibles of people’s brain maps may explain some of the dramatic differences in experienced imagery strength that Galton’s respondents reported. But they cannot explain one of Galton’s most intriguing observations: some people do not experience mental imagery at all. He wrote, “To my astonishment, I found that the great majority of the men of science to whom I first applied protested that mental imagery was unknown to them, and they looked on me as fanciful and fantastic in supposing that the words ‘mental imagery’ really expressed what I believed everybody supposed them to mean. They had no more notion of its true nature than a colour-blind man, who has not discerned his defect, has of the nature of colour. They had a mental deficiency of which they were unaware, and naturally enough supposed that those who affirmed they possessed it, were romancing.”
Modern studies have upheld Galton’s observation: some people simply do not experience mental imagery at all. But why might that be? Science hasn’t yet arrived at a comprehensive answer to this question, but we do have intriguing clues. One thing is clear: insofar as the experience of mental imagery depends on activation of sensory brain maps like the V1 visual map, damage that affects a person’s perception also affects their mental imagery. When strokes, head trauma, or other misfortunes damage visual areas of a person’s brain and disrupt their visual perception, their visual imagery is affected in a similar way. For example, people can lose the ability to see color after brain damage to a particular region of the visual cortex. These patients can still see the world, but it is now in shades of gray. They can also still generate clear mental images, but those images too have been drained of all color. Other people with brain damage that destroyed the FFA face zone lose the ability to recognize the faces around them and to generate mental images of specific faces. Even momentarily disrupting neural activity in a healthy person’s V1 map with bursts of transcranial magnetic stimulation makes them temporarily perform worse on both tests that require visual perception and tests that require mental imagery.
Although damage that wipes out perception tends to wipe out imagery, the reverse is not necessarily true. One example is a patient whom I will call Michael. This sixty-five-year-old man experienced what seemed to be a minor complication during a procedure involving his coronary arteries. Before the surgery, Michael often visualized buildings for his job and visualized faces and events at night before falling asleep. After it, he found that he could no longer visualize images while awake nor see them in his dreams at night. Tests of his visual and neurological function came back normal. He could see just fine. He could even remember things that he’d seen. But he could not call the images to mind. As he put it, “I can remember visual details, but I can’t see them . . . I can’t explain that . . . From time to time I do miss being able to see.” Michael’s condition—the inability to generate mental images—only recently received a name: aphantasia.
Scientists studied Michael’s brain using functional MRI. When he viewed pictures of famous faces, he showed normal activity in the FFA face zone and in visual maps, including V1. But when he was asked to visualize famous faces, these areas showed substantially less activity than they did for normal visualizers who performed the same task. The visual areas of Michael’s brain were working and apparently normal, but for some reason they were not being used for imagery. Somehow, the spark that should have set these areas into action had been extinguished.
Whereas Michael lost his ability to visualize imagery after a surgery, others, including Galton’s buddies in science, report never having had that ability to begin with. These people are not visually or cognitively impaired. They are perfectly capable of remembering and being creative. Many of them report that they didn’t learn until their teens or twenties that other people actually see things in the mind’s eye.
All of this suggests that mental imagery is not an essential capacity for human thought. Although many of us harness our brain’s machinery for visual perception and co-opt it for remembering, imagining, and mentally manipulating information, the brain can also solve those particular problems in other ways. Mental imagery is simply a neural strategy—one clever trick for helping you wring as many useful capacities as possible out of your finite brain. Your V1 visual map is wonderful at representing visual information in the service of visual perception. Why shouldn’t your brain also use it to represent visual information in service of recollection, working memory, story comprehension, and imagination?
The idea that parts of the brain can be co-opted to carry out different but related functions has been called neural reuse. This term makes it sound as if we are taking a discarded, unused thing and finding a new purpose for it, but that is not the case. Rather, we are finding more uses for a thing that is already quite actively being used. In that sense, it might be more accurate to call it opportunistic neural cohabitation. In essence, it is like taking a desk that one person is happily using and adding two or three more people to work at the same desk at the same time. If you have a limited number of desks but no shortage of workers, tripling up in this way might be a boon; you are now getting more out of that desk. But of course, there is a downside too, because your three workers are now competing for space and resources. The more productive one worker is, the longer the others are left waiting for tools and space. That is why seeing or hearing something new destroys working memory for patterns or words, respectively; there just aren’t enough neural keyboards and staplers to go around.
The fact that mental imagery and perception interfere with each other might help explain why people vary so much in their use of mental imagery. Although there are clearly some benefits to making mental images, there are very likely costs too. By forcing perception to share a workspace with mental imagery, people who make imagery may give up some of their ability to purely and accurately perceive. In that light, aphantasia is not a “defect,” as Galton described it. Instead, it is a different way of using the brain, which confers different benefits and drawbacks for perceiving, thinking, and remembering. In short, although opportunistic strategies like mental imagery help us wring new abilities out of our physically limited brains, they are still subject to the stiff competition and tough tradeoffs that characterize every aspect of brain maps and neural representation.
Thanks to intrepid scientists and recent technologies, we now have real purchase on the question “What is mental imagery?” But mental imagery is not the only product of the mind that should, by all rights, lie beyond the reach of science. Another familiar yet ineffable feature of your mind is its capacity to attend. Attention is invisible, elusive, and hard to describe. And yet, like mental imagery, attention operates within your brain maps and can be detected there with brain scans.
While you have been reading this book, you have been receiving my words and, from them, constructing meaning. There are countless other things you could have been doing instead: noticing the pressure of your shoe against your big toe, hearing the trill of a nearby bird, or pondering that ever-present question, What’s for dinner? You might now stop reading and do any one of those things, but if you do, it will take something away from my words and their meaning. Some thing is bequeathed to either this book or to dinner, or thinly spread like an insufficient condiment across them both. That thing, so familiar to us from our youngest years, is attention. Whether out of interest, surprise, or the threat of a failing grade, you invest your attention in one object, activity, or place, only to call in the loan and divert those mental funds to something else entirely. You do this continuously and with ease. You are a veritable Wall Street day trader when it comes to allocating and reallocating this ethereal cognitive resource. But what is attention, and how does your brain make it?
William James, an influential American psychologist of the nineteenth century, described attention like this: “Everyone knows what attention is. It is the taking possession by the mind, in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought. Focalization, concentration, of consciousness are of its essence.”
Perhaps everyone does know what attention is. But let’s imagine for a moment a person or creature who doesn’t. Imagine a species from another planet, with brains that record and process all of the sensory details of the world around them faithfully and in their entirety—a sort of panoramic multisensory video recording of every sight, sound, and sensation. To them, James’s definition would seem inscrutable. How could our minds take possession of anything? How could they refrain from registering events or objects that are right before our very eyes? What exactly is concentrated? What becomes focalized, and where, and how? The very fabric of their experience would seem to be so wholly different from ours that we might declare it futile to attempt explaining attention.
But let us try. One way to explain attention might be to describe the observable effects that attention has on our behavior. What does paying attention to a place or an object do for us? In a nutshell, it makes us better at sensing certain things—things that we are interested in or searching for. We are better able to perceive and detect things—dim shapes, faint sounds, and the like—when we attend to them. We also become faster at finding them. Attending to something is like using binoculars to see distant objects or infrared goggles to see people at night; it boosts the ability to see that something is there and to discover what that thing might be.
But there is a flip side to attention. In making us better at perceiving a particular target, it leaves us worse at perceiving almost anything else. It seems to be a finite resource that must be strategically moved from one moment to the next like troops on a battlefield. James and many others argue that one reason attention is so useful is because it shields us from the mountains of irrelevant information that our senses would otherwise deliver to us. Without attention, they argue, we would be overwhelmed and overloaded. And yet this wouldn’t be true for our space-traveling acquaintance. Our friend is perfectly happy to drink from the fire hose of panoramic sensory experience. There is no reason why a mind couldn’t perceive and process all of that information simultaneously. And yet it is patently clear that our minds cannot. Why is that?
To answer that question, recall the tough tradeoffs inherent in brain evolution. Your brain mustn’t be too big or too heavy, or demand too much fuel. Brain maps are one of nature’s solutions to this problem—a way to reap the most from a finite brain. But brain maps aren’t enough to get you there. And so our brain maps are warped by magnification, bequeathing vast expanses of territory to represent inputs from favored regions of the body or the visual field or the sound-frequency spectrum, while giving short shrift to others. And yet even with warped brain maps, the wide world is still too busy and vast for us to manage at any single point in time. If we hoped to receive and process information about all locations and objects and senses from all of our maps all at once, our brains would have to be substantially larger. Implausibly so. Our alien friend could manage this only if he possessed an enormous head, few or imprecise senses, or perhaps all of the above. Attention is a brilliant solution to these constraints. We can have small heads and sharp senses, but we can dynamically boost the processing of certain things that are important to us, moment by moment, at the expense of those things that are (for the moment, anyway) not.
It is easy to see attention work its fickle magic on all of the brain maps that we already know and love. When you attend to reading the words written on this page, neural activity in the foveal region of your V1 visual map is high. So too is activity in the zone of your object map devoted to processing letters. But when you shift your attention to how your shoe presses against your right big toe, activity in visual and object maps drops and activity in the big-toe region of your left S1 touch map rises. Do it now; shift your attention from what you see to what you feel. Notice that the same patterns of light and dark were falling on your retinas and the same patterns of compression impinged on the skin of your toes before and after you shifted your attention. The change in intention that came from within you caused a major change in how quickly neurons in your V1 and S1 maps fired, alongside major changes in what you perceived. Just as you can willfully call a mental image to mind by using your brain maps, you can willfully amp up or tamp down the activity in your various brain maps by allocating attention.
Think of how effortlessly and yet profoundly that act changed the nature of your experience—from one that is primarily visual, reflecting a tableau of ricocheting photons, to one that is primarily tactile, reflecting the mechanical pressure of molecules in your skin and molecules in your shoe repelling one another. Now, if you stop to listen to the ambient sounds around you, you will ramp up the firing of neurons in your auditory cortex and simultaneously become aware of sounds—perhaps a clock ticking or a bird chirping—that you may not have noticed before. Tune in to each of your senses in turn. For each, you will find a panoply of sensations awaiting you, unnoticed until you sought them out.
Attention does not just momentarily privilege a particular sense over another. Even within a sensory modality, it can favor the processing of things in certain places. Here too we can see attention at work in our brain maps. For example, when you attend to items at your center of gaze, activity is high in the foveal part of the V1 visual map, regardless of what else might be happening in your visual field. Now let’s say that you keep your gaze glued where it is, but you shift your attention outward, away from where you are pointing your eyes. You have probably employed this kind of stealthy attention when you wanted to look at someone without their knowing it. When you shift visual attention in this way, neural activity drops in the foveal part of V1 and ramps up in the corresponding peripheral region of the V1 map.
Attention can also privilege the processing of certain types of things. If you want to locate a face in a complex visual scene, you can attend to faces, wherever they may be. Attending to faces boosts activity in face-preferring zones of the object map, such as the FFA face zone. It will also help you locate the face much faster. Ditto if you want to locate a building in a picture: you attend to buildings and simultaneously boost activity in the PPA place zone. If you want to detect a specific speech sound, you can ramp up corresponding regions of your auditory cortex. If you want to detect whether a fluid tastes sweet, you can increase activity in regions of your taste cortex.
When you zoom in and try to understand exactly how attention affects the firing of individual neurons within these brain maps, the picture becomes substantially more complex. In some cases, directing attention toward a feature or location that a particular cell prefers may simply and straightforwardly increase how often that cell fires. But in many cases, attention has more nuanced effects. It can boost a neuron’s sensitivity, making the cell more likely to fire for the faintest glimmer of a target. Or it can turbocharge its firing rate only when that rate is already high—in a sense, amplifying its active state and making the difference between Nope, nothing and Yes, something more obvious to the rest of the brain.
Willfully attending to something has a remarkable impact on sensory brain maps and actual perception. But what triggers those changes? If attention is like legions of troops being strategically deployed and moved between various senses and parts of sensory maps, who is the general commanding those troops? At this point, we know several key figures in the chain of command—and all of them contain their own maps. Recall the spatial maps of salience or intent in the parietal cortex, which combine information from multiple senses and coordinate systems. These maps collect information from the senses about what and where are important right now. They relay crucial information to the motor cortex, so that you can act upon or react to these important things or people. But they also send this information back to the sensory areas, boosting and hushing regions of the sensory maps as needed in the moment. The motor cortex is also important for directing attention. The frontal eye fields are regions in the motor cortex that contain their own visual map. The neurons within the frontal eye fields generate eye movements and are responsible for directing your gaze toward events and objects of interest in your environment. But they also direct attention-related activity in other parts of the brain, including sensory brain maps.
Although there is no single area of the brain that plays puppet master when it comes to directing attention, the major players are poised at the intersection of action and perception. Attention knows what is relevant for behavior at this moment in time because it is listening to your motor system. That is the beauty of your interconnected brain: perception is always shaping action even as action is perpetually, fundamentally shaping perception.
Of all the thoughts and capacities that arise from the mind, attention and mental imagery seem the most personal and ethereal. As such, it is astonishing how much we now understand about what these phenomena actually are and how they play out in our brain maps.
Another grand challenge for neuroscientists has been to understand how the brain represents concepts and meaning. Abstract concepts like number, time, love, and failure are divorced from the here and now. They cannot be seen or touched, but they still must be represented, understood, and discussed. Remarkably, brain maps have a role to play when it comes to representing these intangible notions as well.