Brain Rule #9
Vision trumps all other senses.
WE DO NOT SEE with our eyes. We see with our brains.
The evidence lies with a group of 54 wine aficionados.
Stay with me here. To the untrained ear, the vocabularies that wine tasters use to describe wine may seem pretentious, more reminiscent of a psychologist describing a patient. (“Aggressive complexity, with just a subtle hint of shyness” is something I once heard at a wine-tasting soirée to which I was mistakenly invited—and from which, once picked off the floor rolling with laughter, I was hurriedly escorted out the door.)
These words are taken very seriously by the professionals, however. A specific vocabulary exists for white wines and a specific vocabulary for red wines, and the two are never supposed to cross. Given how individually we each perceive any sense, I have often wondered how objective these tasters actually could be. So, apparently, did a group of brain researchers in Europe. They descended upon ground zero of the wine-tasting world, the University of Bordeaux, and asked: “What if we dropped odorless, tasteless red dye into white wines, then gave it to 54 wine-tasting professionals?” With only visual sense altered, how would the enologists now describe their wine? Would their delicate palates see through the ruse, or would their noses be fooled? The answer is “their noses would be fooled.” When the wine tasters encountered the altered whites, every one of them employed the vocabulary of the reds. The visual inputs overrode their other highly trained senses. Folks in the scientific community had a field day. Professional research papers were published with titles like “The Color of Odors” and “The Nose Smells What the Eye Sees.” That’s about as much frat-boy behavior as prestigious brain journals tolerate, and you can almost see the wicked gleam in the researchers’ eyes. Studies such as these point to the nuts and bolts of Brain Rule #9. Visual processing doesn’t just assist in the perception of our world. It dominates the perception of our world.
Not like a camera
Many people think that the brain’s visual system works like a camera, simply collecting and processing the raw visual data provided by our outside world. Seeing seems effortless, 100 percent trustworthy, capable of providing a completely accurate representation of what’s actually out there. Though we are used to thinking about our vision in such reliable terms, nothing in that last sentence is true. The process is extremely complex, seldom provides a completely accurate representation of our world, and is not 100 percent trustworthy. We actually experience our visual environment as a fully analyzed opinion about what the brain thinks is out there.
It starts with the retina, vying for the title of amateur filmmaker. We used to think the retina acted like a passive antenna in an automated process: First, light (groups of photons, actually) enters our eyes, where it is bent by the cornea, the fluid-filled structure upon which your contacts normally sit. The light travels through the eye to the lens, where it is focused and allowed to strike the retina, a group of neurons in the back of the eye. The collision generates electric signals in these cells, and the signals travel to the back of the brain via the optic nerve for analysis. But, it turns out, the retina isn’t just waving through a series of unaltered electric signals. Instead, specialized nerve cells deep within the retina interpret the patterns of photons, assemble the patterns into a collection of “movies,” and then send these movies for analysis. The retina, it seems, is filled with teams of tiny Martin Scorseses. These movies are called tracks.
Tracks are coherent, though partial, abstractions of specific features of the visual environment. One track appears to transmit a movie you might call Eye Meets Wireframe. It is composed only of outlines, or edges. Another makes a film you might call Eye Meets Motion, processing only the movement of an object (and often in a specific direction). Another makes Eye Meets Shadows. There may be as many as 12 of these tracks operating simultaneously in the retina, sending off interpretations of specific features of the visual field. This new view is quite unexpected. It’s like discovering that the reason your TV gives you feature films is that your cable is infested by a dozen independent filmmakers, hard at work creating the feature while you watch it.
Rivers of visual information
These movies now stream out from the optic nerve, one from each eye, and flood the thalamus, that egg-shaped structure in the middle of our heads that serves as a central distribution center for most of our senses. If these streams of visual information can be likened to a large, flowing river, the thalamus can be likened to the beginning of a delta. Once the information leaves the thalamus, it travels along increasingly divided neural streams. Eventually, thousands of small neural tributaries will be carrying parts of the original information to the back of the brain. (Put your hand on the back of your head. Your palm is now less than a quarter of an inch away from the visual cortex, the area of the brain that is currently allowing you to see these words.) The information drains into a large complex region within the occipital lobe called the visual cortex.
Once they reach the visual cortex, the various streams flow into specific parcels. There are thousands of lots, and their functions are almost ridiculously specific. Some parcels respond only to diagonal lines, and only to specific diagonal lines (one region responds to a line tilted at 40 degrees, but not to one tilted at 45 degrees). Some process only the color information in a visual signal; others, only edges; others, only motion.
This means you can damage the region of the brain in charge of, say, motion, and get an extraordinary deficit. You’d be able to see and identify objects quite clearly, but not tell whether the objects are stationary or moving. This happened to a patient known to scientists as L.M. It’s called cerebral akinetopsia, or motion blindness. L.M. perceives a moving object as a progressive series of still snapshots—like looking at an animator’s drawings one page at a time. This can be quite hazardous. When L.M. crosses the street, for example, she can see a car, but she does not know if it is actually coming at her.
L.M.’s experience illustrates just how modular visual processing is. And if that was the end of the visual story, we might perceive our world with the unorganized fury of a Picasso painting—a nightmare of fragmented objects, untethered colors, and strange, unboundaried edges. But that’s not what happens, because of what takes place next. The brain reassembles the scattered information. Individual tributaries start recombining, merging, pooling their information, comparing their findings, and then sending their analysis to higher brain centers. The centers gather these hopelessly intricate calculations from many sources and integrate them at an even more sophisticated level. Higher and higher they go, eventually collapsing into two giant streams of processed information. One of these, called the ventral stream, recognizes what an object is and what color it possesses. The other, termed the dorsal stream, recognizes the location of the object in the visual field and whether it is moving.
“Association cortices” do the work of integrating the signals. They associate—or, better to say, reassociate—the balkanized electrical signals. Then you see something. So the process of vision is not as simple as a camera taking a picture. The process is more complex and more convoluted than anyone could have imagined. There is no real scientific agreement about why this disassembly and reassembly strategy occurs.
Complex as visual processing is, things are about to get worse.
You’re hallucinating right now
You might inquire whether I had too much to drink if I told you right now that you were actively hallucinating. But it’s true. At this very moment, while reading this text, you are perceiving parts of this page that do not exist. Which means you, my friend, are hallucinating. I am about to show you that your brain actually likes to make things up, that it is not 100 percent faithful to what the eyes broadcast to it.
Blind spots
There is a region in the eye where retinal neurons, carrying visual information, gather together to begin their journey into deep brain tissue. That gathering place is called the optic disk. It’s a strange region, because there are no cells that can perceive sight in the optic disk. It is blind in that region—and you are, too. It is called the blind spot, and each eye has one. Do you ever see two black holes in your field of view that won’t go away? That’s what you should see. But your brain plays a trick on you. As the signals are sent to your visual cortex, the brain detects the presence of the holes, examines the visual information 360 degrees around the spot, and calculates what is most likely to be there. Then, like a paint program on a computer, it fills in the spot. The process is called “filling in,” but it could be called “faking it.” Some scientists believe that the brain simply ignores the lack of visual information, rather than calculating what’s missing. Either way, you’re not getting a 100 percent accurate representation.
Dreams during the night—or day
It should not surprise you that the brain possesses an imaging system with a mind of its own. Proof is as close as your most recent dream. (There’s a hallucination for you.) Actually, the visual system is even more of a loose cannon than that. Millions of people suffer from a phenomenon known as the Charles Bonnet Syndrome. Most who have it keep their mouth shut, however, and perhaps with good reason. People with Charles Bonnet Syndrome see things that aren’t there.
Everyday household objects suddenly pop into view. Or unfamiliar people unexpectedly appear next to them at dinner. Neurologist Vilayanur Ramachandran describes the case of a woman who suddenly—and delightfully—observed two tiny policemen scurrying across the floor, guiding an even smaller criminal to a matchbox-size van. Other patients have reported angels, goats in overcoats, clowns, Roman chariots, and elves. The illusions often occur in the evening and are usually quite benign. Charles Bonnet Syndrome is common among the elderly, especially among those who previously suffered damage somewhere along their visual pathway. Interestingly, almost all of the patients know that the hallucinations aren’t real.
A camel in each eye
Besides filling in our blind spots and creating bizarre dreams, the brain has another way of participating in our visual experience. We have two eyes, each taking in a full scene, yet the brain creates a single visual perception. Since ancient times, people have wondered why. If there is a camel in your left eye and a camel in your right eye, why don’t you perceive two camels? Here’s an experiment to try that illustrates the issue nicely.
1) Point your left index finger to the sky. Touch your nose and then stretch your left arm out.
2) Point your right index finger to the sky. Touch your nose and then move your finger about six inches away from your face.
3) Both fingers should be in line with each other, directly in front of your nose.
4) Now speedily wink your left eye and then your right one. Do this several times, back and forth. Your right finger will jump to the other side of your left finger and back again. When you open both eyes, the jumping will stop.
This little experiment shows that the two images appearing on each retina always differ. It also shows that both eyes working together give the brain enough information to create one stable picture. One camel. Two non-jumping fingers. How?
The brain interpolates the information coming from both eyes. Just to make things more complicated, each eye has its own visual field, and they project their images upside down and backward. The brain makes about a gazillion calculations, then provides you its best guess. And it is a guess. You can actually show that the brain doesn’t really know where things are. Rather, it hypothesizes the probability of what the current event should look like and then, taking a leap of faith, approximates a viewable image. What you experience is not the image. What you experience is the leap of faith.
The brain does this because it needs to solve a problem: The world is three-dimensional, but light falls on the retina in a two-dimensional fashion. The brain must deal with this disparity if it is going to portray the world with any accuracy. To make sense of it all, the brain is forced to start guessing. Upon what does the brain base its guesses, at least in part? Experience with past events. After inserting numerous assumptions about the visual information (some of these assumptions may be inborn), the brain then offers up its findings for your perusal. Now you see one camel when there really is only one camel—and you see its proper depth and shape and size and even hints about whether it will bite you.
Far from being a camera, the brain is actively deconstructing the information given to it by the eyes, pushing it through a series of filters, and then reconstructing what it thinks it sees. Or what it thinks you should see. All of this happens in about the time it takes to blink your eyes. Indeed, it is happening right now. If you think the brain has to devote to vision a lot of its precious thinking resources, you are right on the money. Visual processing takes up about half of everything your brain does, in fact. This helps explain why professional wine tasters toss aside their taste buds so quickly in the thrall of visual stimuli. And why vision affects other senses, too.
Vision trumps touch, not just smell and taste
Amputees sometimes continue to experience the presence of their limb, even though the limb no longer exists. In some cases, the limb is perceived as frozen into a fixed position. Sometimes the person feels pain in the limb. Studies of people with phantom limbs demonstrate the powerful influence vision has on our other senses.
In one experiment, an amputee with a “frozen” phantom arm was seated at a table upon which had been placed a lidless box divided in half. The box had two portals in the front, one for the arm and one for the stump. The divider was a mirror on both sides. So the amputee could view a reflection of either his functioning hand or his stump. When the man looked down into the box, he could see his right arm present and his left arm missing. But when he looked at the reflection of his right arm in the mirror, he saw what looked like another arm. Suddenly, the phantom limb on the other side of the box “woke up.” If he moved his normal hand while gazing at its reflection, he could feel his phantom move, too. And when he stopped moving his right arm, he felt his missing left arm stop also. The addition of visual information began convincing his brain of a miraculous rebirth of the absent limb.
A picture really is worth a thousand words
One way we can measure the dominance of vision is to look at its effect on learning. Researchers study this using two types of memory.
The first is called recognition memory, which underlies the concept of familiarity. We often deploy recognition memory when looking at old family photographs. Maybe you see a photo of an aunt not remembered for years. You don’t necessarily recall her name, or the photo, but you still recognize her as your aunt. With recognition memory, you may not recall certain details surrounding whatever you see, but as soon as you see it, you know that you have seen it before.
The second involves working memory. Explained in greater detail in the Memory chapter, working memory is that collection of temporary storage buffers with fixed capacities and frustratingly short life spans. Visual short-term memory is the slice of that buffer dedicated to storing visual information. Most of us can hold the memory of about four objects at a time in that buffer, so it’s a pretty small space. And it appears to be getting smaller. As the complexity of objects in our world increases, we are capable of remembering fewer objects over our lifetimes. Evidence also suggests that the number of objects and complexity of objects are engaged by different systems in the brain—turning the whole notion of short-term capacity, if you will forgive me, on its head. These limitations make it all the more remarkable that vision is probably the best single tool we have for learning anything.
When it comes to both recognition memory and working memory, pictures and text follow very different rules. Put simply, the more visual the input becomes, the more likely it is to be recognized—and recalled. It’s called the pictorial superiority effect. Researchers have known about it for more than 100 years. (This is why we created a series of videos and animations of the Brain Rules at www.brainrules.net, making this book just one part of a multimedia project.)
The pictorial superiority effect is truly Olympian. Tests performed years ago showed that people could remember more than 2,500 pictures with at least 90 percent accuracy several days later, even though subjects saw each picture for about 10 seconds. (This is recognition memory, not working memory, at work.) Accuracy rates a year later still hovered around 63 percent. In one paper, picture recognition information was reliably retrieved several decades later. Sprinkled throughout these experiments were comparisons with text or oral presentations. The usual result was “picture demolishes them both.” It still does. Text and oral presentations are not just less efficient than pictures for retaining certain types of information; they are far less efficient. If information is presented orally, people remember about 10 percent, tested 72 hours after exposure. That figure goes up to 65 percent if you add a picture.
Why is text less efficient than pictures? Because, it turns out, the brain sees words as lots of tiny pictures. A word is unreadable unless the brain can separately identify simple features in the letters. Instead of words, we see complex little art-museum masterpieces, with hundreds of features embedded in hundreds of letters. Like an art junkie, our brains linger at each feature, rigorously and independently verifying it before moving to the next. So reading creates a bottleneck in comprehension. To our cortex, surprisingly, there is no such thing as words.
That’s not necessarily obvious. After all, the brain is as adaptive as Silly Putty. Given your years of reading books, writing email, and sending text messages, you might think your visual system could be trained to recognize common words without slogging through tedious additional steps of letter-feature recognition. But that is not what happens. No matter how experienced a reader you become, your brain will still stop and ponder the individual features of each letter you read—and do so until you can’t read anymore.
By now, you can probably guess why this might be. Our evolutionary history was never dominated by books or email or text messages. It was dominated by trees and saber-toothed tigers. Vision means so much to us because most of the major threats to our lives in the savannah were apprehended visually. Ditto with most of our food supplies. Ditto with our perceptions of reproductive opportunity.
The tendency is so pervasive that, even when we read, most of us try to visualize what the text is telling us. “Words are only postage stamps delivering the object for you to unwrap,” George Bernard Shaw was fond of saying. A lot of brain science now backs him up.
Vision is king from Day One
Babies come with a variety of preloaded software devoted to visual processing. We can determine what babies are paying attention to simply by watching them stare at their world. The importance of a baby’s gazing behavior cannot be underestimated.
You can see this for yourself (if you have a baby nearby). Tie a ribbon around the baby’s leg. Tie the other end to a mobile. At first she seems to be randomly moving her limbs. Soon, however, the infant learns that if she moves one leg, the mobile turns. She begins happily—and preferentially—moving that leg. Bring back the same mobile the next week, and the baby will move the same leg. Show the baby a different mobile, and she won’t move the leg. That’s what scientists found when they did this experiment. The baby is paying the most attention to the visual aspects of the mobiles. Since the mobiles don’t look the same, there’s not much reason to assume they would act the same. Babies use these visual cues even though nobody taught them to do so. This illustrates the importance of visual processing to our species.
Other evidence points to the same fact. Babies display a preference for patterns with high contrast. They seem to understand the principle of common fate: Objects that move together are perceived as part of the same object, such as stripes on a zebra. They can discriminate human faces from nonhuman equivalents and seem to prefer the human faces. They possess an understanding of size related to distance—that if an object is getting closer (and therefore getting bigger), it is still the same object. Babies can even categorize visual objects by common physical characteristics. The dominance of vision begins in the tiny world of infants.
It also shows up in the even tinier world of DNA. Our sense of smell and color vision are fighting each other for evolutionary control, for the right to be consulted first whenever something on the outside happens. And vision is winning. In fact, about 60 percent of our smell-related genes have been permanently damaged in this neural arbitrage, and they are marching toward obsolescence at a rate fourfold faster than any other species sampled. The reason for this decommissioning is simple: The visual cortex and the olfactory cortex take up a lot of neural real estate. In the crowded zero-sum world of the sub-scalp, something has to give. Does this mean that we’ll permanently lose our sense of smell or that our heads are no longer getting bigger? Check back in several hundred thousand years. The evolutionary forces that actively selected against smell are not still in full force today. But what forces are replacing them is an active area of debate.
Whether looking at behavior, cells, or genes, we can observe how important the visual sense is to the human experience. Striding across our brain like an out-of-control superpower, giant swaths of biological resources are consumed by it. In return, our visual system creates movies, generates hallucinations, and consults with previous information before allowing us to see the outside. It happily bends the information from other senses to do its bidding and, at least in the case of smell, seems to be caught in the act of taking over.
When it comes to applying this knowledge in your own daily life, is there any point in trying to ignore the vision juggernaut? You don’t have to look any further than the wine experts of Bordeaux for the answer.
More ideas
The best visuals for learning
What kind of pictures best grab attention and thus transfer information? We pay lots of attention to color. We pay lots of attention to orientation. We pay lots of attention to size. And we pay special attention if the object is in motion. Indeed, most of the things that threatened us in the Serengeti moved, and the brain has evolved unbelievably sophisticated trip wires to detect motion. We even have specialized regions to distinguish when our eyes are moving versus when our world is moving. These regions routinely shut down perceptions of eye movement in favor of the environmental movement.
That said, we need more research into practical applications. The pictorial superiority effect is a well-established fact for certain types of classroom material, but not for all material. Data are sparse. Do pictures communicate conceptual ideas such as “freedom” and “amount” better than, say, a narrative? Are language arts better represented in picture form or using other media? It’s unclear.
Include video or animation
I owe my career choice to Donald Duck. I am not joking. I even remember the moment he convinced me. I was 8 years old at the time, and my mother trundled the family off to a showing of an amazing 27-minute animated short called Donald in Mathmagic Land. Using visual imagery, a wicked sense of humor, and the wide-eyed wonder of an infant, Donald Duck introduced me to math. Got me excited about it. From geometry to football to playing billiards, the power and beauty of mathematics were made so real for this nerd-in-training, I asked if I could see it a second time. My mother obliged, and the effect was so memorable, it eventually influenced my career choice. I now have a copy of those valuable 27 minutes in my own home and regularly inflict it upon my poor children. Donald in Mathmagic Land won an Academy Award for best animated short of 1959. It also should have gotten a Teacher of the Year award. The film illustrates—literally—the power of the moving image in communicating complex information to students.
Animating presentations is another way to capture the importance not only of color and placement but also of motion. The basics are not hard to learn. With today’s software, anybody who knows how to draw a square and a circle can create simple animations. Simple two-dimensional pictures are quite adequate; studies show that if the drawings are too complex or lifelike, they can distract from the transfer of information.
Communicate with pictures more than words
“Less text, more pictures” were almost fighting words in 1982. They were used derisively to greet the arrival of USA Today, a brand-new type of newspaper with, as you know, less text, more pictures. Some predicted the style would never work. Others predicted that if it did, the style would spell the end of Western civilization as the newspaper-reading public knows it. The jury may be out on the latter prediction, but the former has a powerful and embarrassing verdict. Within four years, USA Today had the second-highest readership of any newspaper in the country, and within 10 years, it was number one. It still is.
What happened? Pictorial information may be initially more attractive to consumers, in part because it takes less effort to comprehend. Because it is also a more efficient way to glue information to a neuron, there may be strong reasons for entire marketing departments to think seriously about making pictorial presentations their primary way of transferring information.
The initial effect of pictures on attention has been tested. Using infrared eye-tracking technology, 3,600 consumers were tested on 1,363 print advertisements. The conclusion? Pictorial information was superior in capturing attention—independent of its size. Even if the picture was small and crowded with lots of other nonpictorial elements close to it, the eye went to the visual.
Toss your PowerPoint presentations
The presentation software called PowerPoint has become ubiquitous, from work meetings to college classrooms to conferences. What’s wrong with that? They’re mostly text, even though they don’t have to be. A typical PowerPoint business presentation has nearly 40 words per slide. Please, do two things: (1) Burn your current PowerPoint presentations. (2) Make new ones. Then see which one works better.
Brain Rule #9
Vision trumps all other senses.
• Vision is by far our most dominant sense, taking up half of our brain’s resources.
• What we see is only what our brain tells us we see, and it’s not 100 percent accurate.
• The visual analysis we do has many steps. The retina assembles photons into movie-like streams of information. The visual cortex processes these streams: some areas registering motion, others registering color, etc. Finally, we recombine that information so that we can see.
• We learn and remember best through pictures, not through written or spoken words.
