Making Sense Out of a Noisy Environment
“The world is noisy and messy. You need to deal with the noise and uncertainty.” —Daphne Koller, computer scientist
More than fifty years ago, while some of the split-brain studies were being initiated, Dutch psychologist Willem Levelt published a long paper entitled “On Binocular Rivalry,” about a phenomenon that had been known since the nineteenth century. It occurs when each of a person’s two eyes are viewing strikingly different imagery or stimuli. The person observing the images can perceive only one of the images at a time, and perception of the image by the brain switches back and forth between the two images from the two eyes. Related to this phenomenon but somewhat different from it are optical illusions (box 14.1).
Rivalry also exists for some of the other senses. Several decades ago, researchers reported on a novel hearing illusion that they suggested was the result of auditory rivalry. The experiments that pinned down this auditory illusion involved right- and left-handed subjects. The choice of these subjects is valid, because handedness apparently indicates which side of the brain is the dominant side, and knowing the dominant side of an individual’s brain is critical to how this auditory illusion is interpreted. A short description of handedness is appropriate here, and since a lot of what we know about handedness has been placed into a genetic context, I will use genetics to explain it. The genetics of some traits in humans are pushovers—they can be pinned on a single gene region that controls the expression of the trait, such as color blindness. Other traits, such as height or weight, are tougher as it turns out because multiple genes control them. Over the years many candidate genes have been proposed as controllers of handedness, but none with much certainty. Modern genome-wide association studies using whole genomes of thousands of people have been unable to pin down candidate genes, and even the tried-and-true identical twin approach has failed to distinguish candidate genes. The inability to pinpoint candidate genes suggests that many genes are involved in the expression of handedness in human populations.
One classic example of an optical illusion is the juxtaposition of two identical, solid white silhouettes of a man or woman nose to nose (see fig. 14.1; another similar illusion is also shown in the figure). The space between the two profiles is filled in with black to form an urn. All kinds of illusions like this one are obvious reminders that humans never perceive both images at the same time; rather, our perception of the two images shifts as the brain is forced to perceive one or the other. We simply cannot see both images at the same time. It is appropriate that the phenomenon was discovered during the invention of the stereoscope. Although binocular rivalry has been recognized for almost two centuries, we still do not know exactly how it works in the brain.
Figure 14.1. Two classic black-and-white optical illusions. Do you see vases or faces in the left side? Do you see sexy or sax on the right?
Two important inferences come out of the genetic studies of handedness that are relevant to brain sidedness. First, because the trait is genetically complex and probably the result of additive effects from many genes, expression of the trait may be the result of the effects of genes accumulating to allow for a bias for right-handedness. For those who do not develop the bias, whether they become left-handed is a matter of chance. In other words, there is no so-called left-handed gene. The second point is that perhaps there is more than a single genetic way to develop this bias toward right-handedness. Nevertheless, if someone is right-handed, then most likely the left side of his or her brain is dominant, and if they are left-handed, the dominant side of the brain is the right side.
Using this trick, Diana Deutsch took right- and left-handers, placed headphones on them, and blipped sounds at different pitches into their ears. Deutsch exposed her listeners to alternating pitches of very brief (one-fourth of a second) blips of sound at 400 hertz, followed by a blip at 800 hertz, with no gap between the blips. The amplitude of the sound was equal for both pitches. The difference between the left and right ears was such that, when the left ear was hearing 400 hertz, the right was exposed to 800 hertz. These two tones were selected because the lower tone (400 Hz) is easier to hear than the higher tone (800 Hz). If perceived correctly, one should hear the high pitch in one ear and the low pitch in the other ear. And it should shift back and forth from the left ear to the right. It should sound like an alternation between “whoo-hoo, hoo-whoo, whoo-hoo, hoo-whoo,” where the first “hoo” or “whoo” is in the left ear and the second is in the right ear. There should be an undulation between ears.
Surprisingly, most people simply hear the one tone in one ear and the second tone in the other ear, and the tones alternate. So, it sounds like “whoo” (right ear), “hoo” (left ear), “whoo” (right ear), “hoo” (left ear), and so on. A good number of individuals simply heard “whoo” (right ear), “whoo” (left ear), “whoo” (right ear), “whoo” (left ear), and so on. And a small percentage heard a third ghost (nonexistent) tone interspersed with the two real tones. Of the eighty-six subjects in the study, not one perceived the correct pattern of tones or their undulation. But what about handedness? It turns out that right-handers localized the higher tone to the right ear and the lower tone to the left. Even when the earphones delivering the tones were reversed, they still heard the tones this way. Left-handers did not localize the sounds to either ear and settled more or less randomly on the ear that “heard” the higher tone and the ear that “heard” the lower tone.
So, it seems that any structures on our bodies that have two entries for the sensory stimulation, or in other words are bilateral, have this rivalry. Bilateral symmetry evolved hundreds of millions of years ago, and most higher animals are great examples of such symmetry.
What other sensory organs are bilateral? Three come to mind—one obvious and the other two not so much so. The obvious one is the olfactory primary sense organ, or nostrils. One less obvious one is the touch organs placed symmetrically on the left and right sides of the body. And another less obvious one would be the balance organs.
Olfactory or nasal rivalry has recently been examined by exposing one nostril to phenylethyl alcohol (PEA) and the other to n-Butanol, two chemicals that smell quite different. PEA is a pleasant-smelling compound with hints of roses and honey, but n-Butanol has a sharp smell—a little like sniffing a magic marker. Because the olfactory system can adapt to odorants very quickly (in about twenty seconds—neither vision nor hearing have this propensity to adapt), the simple binocular and biauditory experiments discussed earlier were modified to minimize the contribution of adaptation to the results. It turns out that there is also a nasal bilateral rivalry. Specifically, when the nostrils of subjects are presented with these two different smells, only one is detected at a time, and the detection alternates much like the auditory and visual systems from one side to the other.
Processing touch is a little different, even though the tactile sense organs are in general bilaterally symmetrical. But research has revealed that the processing of touch stimulation requires not only receiving and transmitting the location of the tactile stimuli to the brain, but also that the locality be combined with information about the current posture or spatial position of the regions experiencing the tactile stimuli. The latter requirement is fulfilled visually. With crossed hands, the processing starts with the brain assuming the usual orientation of the hands—that is, left hand on the left and right hand on the right. But since the hands are really crossed, this position reverses the perception, and someone given tactile stimuli to the hands in a crossed position will immediately perceive and interpret the reverse orientation of the stimulus. As the brain realizes that the arms are indeed crossed from visual information, the tactile information is effectively remapped. The lag time between getting the orientation of the stimuli initially wrong to remapping and getting it right is a critical measure of how the remapping works. Experiments with crossed fingers indicate a similar mapping deficit, but no bilateral anatomical structures are involved in that response. One critical difference between the response of crossed hands and crossed fingers is that with crossed hands the deficit decreases with longer and longer stimulation times, while crossed fingers do not improve even with stimulation lasting almost to a second. This lack of improvement is interpreted as a basic difference between the bilateral hands and the same-side fingers.
The phenomenon has been named the “crossed hands deficit” (fig. 14.2), and it is also evident when the legs are crossed with respect to stimuli to the legs. Additional rather odd experiments have been accomplished to delve into this response. First, the mix-up in temporal order of the stimulation occurs only from the point of crossing out to the distal tip of the limb being crossed. Any location of the stimuli on the limb proximal to the crossing does not experience the phenomenon. Second, the phenomenon can be corrected by having the subject view uncrossed rubber arms instead of their own crossed flesh-and-blood arms. Potential differences exist between the sexes in how the crossed hands deficit works. Because women have been shown to be more visually dependent than males at a specific test called the rod-and-frame test, female visual dependence in spatial matters might mean that they respond differently than males to the crossed-hands deficit effect.
The rod-and-frame test has been used for more than forty years to assess the perception of verticality as a function of visual orientation. In this test, a rod is pictured within a flat, square frame. Both the frame and the rod can be rotated. When the rod and frame are positioned such that they are both perfectly vertical, the observer perceives the verticality of the bar correctly and perception of the true verticality of the rod is not a problem. An illusion is created with this system by rotating the frame away from being vertical to the observer’s field of vision. What happens is that the perception of what is vertical is mixed up. The rod can be perfectly vertical without the frame, but the presence of the frame makes the rod appear to be tilted. If the tilt of the frame is increased, the illusion exaggerates the tilt of the rod. The test asks the observer to tilt the bar so that it is vertical, and the experimenter can measure the impact of the illusion on different observations and with different tilts of the frame. Women are consistently more reliant on their vision to perceive the illusion and correct for it. Because the test requires the hands, it can be administered with crossed hands as well as with the hands in usual positions. The easiest explanation for this sex difference might reside in the differences in how males and females resolve spatial problems.
Figure 14.2. Crossed hands deficit. When the subject crosses her arms and is stimulated on one or the other hands (B1), the stimulation will appear to emanate from the wrong hand (AE) and vice versa.
So far, we have looked at conflict or rivalry within specific senses. When we start to examine how the senses interact with one another, a whole new set of rivalries and solutions to conflicting signals arise. And even more complex are the conflicts and rivalries created by processing the senses from outside world stimuli with the higher functions of our brains like memory and emotions. A lot of important studies have been accomplished to pin down the multisensory perception mechanisms that exist in our brains. The pathways of the signals in the brain from a single sense such as touch or sight are fairly well understood. These pathways indicate a complex route from the sense organs like the eyes or ears to and through the brain that result in perception and are the best evidence for the crossmodal nature of sensing.
Precise description of these pathways involves knowing the many parts and areas of the brain. I am one of these people who tell the cabdriver, “Just get me there, I don’t care how.” Along the way I see landmarks I recognize, but the specific details of the trip are not so important to me. Sometimes I pay more for my ride than I would care to, but when you think of it, evolution kind of works that way, too. I will take this “just get me there” approach in attempting to describe crossmodality of senses and dispense with a lot of brain anatomy in the process. Before going into too much detail about crossmodality of the senses, I will give away some of the story. There are people who routinely can smell shapes, hear colors, and taste sounds (among other sensory mixing). These people have a rare connection of their senses called synesthesia. It will become very surprising, though, as I describe crossmodality in people not normally considered synesthetes—that is, the majority of you who are reading this book.
With five senses (visual, auditory, tactile, olfactory, and gustatory) and balance (vestibular) thrown in for good measure, there are fifteen different pairwise sensory interactions. The interactions of all of these pairs have not been fully studied and hence understood, and some are better worked out than others. If we start to examine more complex interactions such as the crossmodality of three senses, the subject gets very messy. It would be a long, repetitive chapter if we examined all possible crossmodalities. So here, to make the point of the capacity of the brain to process in crossmodal fashion, we’ll stick with some of the more interesting binary ones. My favorite science center exhibits (and indeed we included one in an exhibition on the brain at the American Museum of Natural History in 2013), concern the visual-auditory crossmodality. In the AMNH experience, you walk up to the exhibit to see a life-size picture of a woman with an umbrella standing on a rainy street corner with rain pouring down on her. The visual stimulus is supplemented by nearby sound, the patter of rain falling on the street. Or is it? As you walk to the back side of the exhibit, the real source of the sound is revealed. It actually comes from bacon sizzling in a skillet!
Psychologists actually have a test that mimics this frying bacon–rainy day illusion, and they use it to tease apart the very source of visual-auditory crossmodality. One called the double-flash illusion is simple: an observer is presented with a single flash of light accompanied by two auditory prompts (beeps) in rapid succession. Most people when presented with this test will perceive two flashes of light instead of one. The time between the two auditory pulses is a critical factor in determining whether the illusion will occur. When the time between flashes is very brief (less than a hundred milliseconds), the subject is more likely to observe the illusion of two flashes. The illusion stops working at a hard cutoff of a hundred milliseconds, and this suggested to researchers that there might be a universal factor in how sight and sound interact in the brain. Roberto Cecere, Geraint Rees, and Vincenzo Romei examined the possibility that visual perception in the real world is the result of an integration of information from multiple senses where information is weighted in real time to produce “a unified interpretation of an event” as a function of the brain and what are known as alpha waves (box 14.2). Cecere and colleagues suggested that the reason that two flashes of light are observed when the auditory beeps are less than a hundred milliseconds apart is that the auditory stimuli are contained within a single alpha-wave phase and the brain rushes to make a conclusion about the visual and auditory rivalry before the next alpha wave passes through the occipital lobe.
BOX 14.2 | OCCIPITAL ALPHA WAVES
A lot is known about occipital alpha waves because they are the most evident physical waveform that can be measured in the brain. This wave function is a useful property of the brain that has been used to measure many properties of neural function. Specifically, the amplitude or strength of the alpha wave has been used to measure involvement of the cortex in tasks. As we have seen in previous chapters, though, waves also have phases, and it turns out that alpha waves have a phase distribution of 100 milliseconds. Researchers hypothesize that each alpha wave that pulses through the brain delivers to the brain specific information that needs to be processed. The information so delivered cannot be augmented until the next wave comes through 100 milliseconds later.
There are two possible reasons for this odd behavior of our brains. Jess Kerlin and Kimron Shapiro suggest that one reason would be an “unfortunate consequence” of the evolutionary process—that being the connection of hearing to seeing in the brain that leads to an artifact of visual perception (the second flash). The second possibility is a much more complex one with respect to thinking about how our brains work. Kerlin and Shapiro invoke a functional reason for this illusion by suggesting that our brains are performing a probabilistic analysis of the information along the lines of one proposed by the Reverend Thomas Bayes, an eighteenth-century English pastor and dabbler in probability. In his own words, Bayes recognized that the probability of something happening is “the ratio between the value at which an expectation depending on the happening of the event ought to be computed, and the value of the thing expected upon its happening.”
This quaint eighteenth-century wording simply suggests that, when assessing the probability of something happening such as a flash of light, one needs to compute the impact of the knowledge of the information about an event based on observations and some prior knowledge of it happening divided by the impact of the thing happening. The contemporary language I have used might not help much, but the bottom line is that prior knowledge of something happening becomes very important when thinking about the world in a Bayesian context. So, what the brain is doing, according to Kerlin and Shapiro, is taking the Bayesian approach by using the prior knowledge that “audio beep equals light flash” to estimate the event probability that a second beep will be accompanied by a flash of light. Apparently, the prior probability of a flash of light accompanying a beep is sufficiently high to convince the brain to interpret every beep as being accompanied by a flash of light. When the beep and the flash are disconnected (more than a hundred milliseconds apart), the prior probability is perceived as low and the brain computes the probability of the event as being low based on this prior probability. In this Bayesian scenario, the brain is continuously making probabilistic statements about events occurring, and when we are left hanging for more information (for example, in the middle of a hundred-millisecond phase of the alpha wave), we use this probability to shape our perception of events. I am not sure which story I like the best. The “unfortunate” version, to me is not as unfortunate as Kerlin and Shapiro suggest. To me it is just evolution, and it shores up how much of a kluge the brain really is. But if the brain is really performing Bayesian analysis to make decisions about perception, that would be very cool indeed.
I have already discussed some of the crossmodal interactions of touch and sight with the crossed hands deficit, but tactile crossmodal interactions with other senses exist. Perhaps the most famous but nevertheless awesome example of tactile auditory crossmodality involves the characters Kiki and Bouba (fig. 14.3). The crossmodality of sounds with shape or texture goes further than Kiki and Bouba. When anthropologists examined the names of creatures in the languages of indigenous peoples, they discovered a surprising association of soft-sounding names using soft consonants with benign small animals and soft-leaved plants and the association of hard-sounding consonants in the names of dangerous or predatory animals and prickly plants. And indeed, as we will soon see, tastes can also be associated crossmodally with word sounds and shapes.
Figure 14.3. Kiki and Bouba. Guess which is which.
Kiki and Bouba are line-drawn sculptures with projections from a basic central body. Kiki differs from Bouba in that it has sharp projections emanating from it, whereas Bouba has blobby amoebalike projections. Kiki is sharp looking, and Bouba is soft and blobby. When presented with pictures or three-dimensional images of these two characters and asked to connect the name to the character, the grand majority of people associate the name Kiki with the sharp-appendaged character and the name Bouba with the amoebalike character. The association is not age dependent, nor is it culturally or language dependent. People of all ages get it, and people from different countries get it. The sharp sounds of the Ks and the soft sounds of the Bs are associated with the shapes of the characters most likely through a crossmodal process in the brain.
Not only do sounds like Kiki and Bouba influence our perception of touch, but sight also has this effect. When we see something that we want to touch and pick up or perhaps walk on, there is a definite advantage to judging its texture before we go ahead and make the attempt. A slippery piece of food is lost when we try to pick it up in a lackadaisical manner, and our necks get broken when we try to walk on a slippery surface and fall. Hence, for adaptive reasons making a tactile judgment from visual stimuli might very well be an important aspect of a primate’s survival. Glossiness of objects can be used to make this visual assessment of texture and has been used as a visual tool for studying the interaction of tactile perception and visual cues for nearly a century. The Ingersoll Glarimeter was first marketed in 1922 to assess the glossiness of paper and was also used to measure glossiness of objects in psychological experiments back then. Researchers realized that glossiness was more complex than just making a single measurement, and studies using glossiness subsided until recently.
It is now known that the perception of glossiness is a complex interaction of tactile expectation and visual stimulus. Using a device that can vary the glossy appearance of objects as well as their slipperiness, Wendy Adams, Iona Kerrigan, and Erich Graf performed experiments to uncover the role of glossiness in the perception of tactile stimuli. These researchers were able to pair gloss with slipperiness across a continuum, with no gloss–no slip on one end and extreme gloss–extreme slip on the other. The results of the study indicate that participants integrate gloss level with the slipperiness level of objects. Specifically, people can detect increases in glossiness when slipperiness is increased. The converse experiment, where glossiness is decreased and slipperiness is increased (the counterintuitive situation), produced a low level of change in perception. In other words, the counterintuitive extreme when presented to a person is not perceived as different as when slippery is presented with glossy. It is as if the brain is integrating the cues from gloss and slipperiness to come to some probabilistic conclusion (hello, Reverend Bayes) that then biases the perception of slipperiness and glossiness.
The interaction of sight, sound, and touch that I have discussed are pretty obvious. But there are also interactions and crossmodalities that involve balance, smell, and taste. For balance, vision plays a huge role, but not always. Remember the spinning figure skaters who sometimes use vision to reorient the information from their vestibular system with respect to head motion (see Chapter 7)? This visual-vestibular crossmodality, however, works only in certain circumstances. Recent experiments can actually tease apart some of the role of vision in balance and how the mechanosensory information from the vestibular system in the inner ear is cross-modulated by vision by using experiments with galvanic vestibular stimulation, or GVS (box 14.3). Normally, if a person is conditioned to swaying, he or she will reweight the information from the vestibular system and rely more on the visual system. The individual would then subconsciously recalculate the position of the body axis that maintains balance based on the downweighted information from the vestibular system and the upweighted information from the visual system. It’s kind of like figure skaters practicing spins over and over to condition themselves to the amazing spinning moves. If they aren’t conditioned and naively spin, this action wreaks havoc on the skaters’ sense of balance until they have practiced the moves repeatedly. The naive spinners’ brains do not have the capacity to receive the reweighted vestibular and visual information, and hence the vestibular system appears to be entirely on its own.
BOX 14.3 | GALVANIC VESTIBULAR STIMULATION AND BALANCE
Using galvanic vestibular stimulation, researchers can either amplify or dampen the response of the vestibular system during head movement when the body sways. When a person is balanced, there is a typical and measurable mechanosensory response of the vestibular organs in the inner ear. By putting helmets that can dampen or enhance the signal of the vestibular system to the brain on study subjects, researchers can induce in the subjects an illusion that while balancing their bodies their heads were moving faster or slower than they really were. The researchers could quantitate both the extent of the response by the subjects with and without vision and the intensity with which people will attempt to compensate for the illusory movement. Sounds like a really cool carnival ride to me, but the results from the study were very illuminating as to how the vestibular system sometimes is on its own even with information from visual cues. Simply put, for naive spinners, vision doesn’t help.
We are bombarded by sensory input all the time, much of it information that is superfluous or junky and what I call neural detritus. If we processed every bit of sensory information to which we were exposed, the brain would be an overworked mess. Shortcuts abound in how we process sensory information, and this is because as the action potential from the external stimulus comes to our brains we have evolved specific ways to sift through the noise and interpret the information coming in. For example, most humans are pretty adept at focusing on a conversation with someone even in a crowded, loud room. On the other hand, often we need to make quick decisions based either on paltry sensory information or on loads of conflicting information. For instance, many optical illusions are the product of the brain simply saying, “I give up, but here is the best I can do.” This chapter focused on several phenomena used to explore this interesting aspect of our ability to sense the outer world and the brain’s capacity to filter out the sensory detritus. This capacity is what makes us able to perceive the outer world as orderly and concisely as possible.
