How Crossmodal Interactions Work in the Brain
“For the sense of smell, almost more than any other, has the power to recall memories and it is a pity that we use it so little.”
—Rachel Carson, ecologist and author
The single-celled organisms that dominate our planet can sense their environment in bits and pieces. Although these organisms are simple compared with the typical vertebrate, they are as fully evolved as any other living thing on the planet. And although their biology cannot be considered perfect (evolution does not strive for perfection), these single-celled organisms have found solutions to the environmental challenges they have faced during their evolution (evolution does strive for solutions). Some single-celled organisms will use very simple mechanisms to sense their environment, and that is good enough for them to survive and pass on the essence of their species (their genomes) to populations of the next generation. Most of the sensing done by single-celled organisms concerns counting how many of their conspecifics are near (quorum sensing), sensing other organisms that they should defend themselves from or consume or exchange DNA with, and other sensing operations that are simple but essential for survival. The perception of a microbe to its outer world, then, is fairly limited and one-dimensional in most microbes. Getting a signal that a predator is near may be the only sense a microbe needs, and in this way many microbes have a one-dimensional perception of the outer world.
As organisms became more and more complex, sensing the environment in different ways became a widely used and successful strategy in evolution. Once an organism has adopted several ways of sensing the environment, the senses can stay independent and send information to the brain without reference to other outer-world stimuli. Or, alternatively, the different senses can evolve to be connected in some way to one another for any number of reasons. Sometimes the connections are made because of structural constraints. If two different senses occupy neural real estate near each other, then connections between them might be made randomly. In other cases where crossmodality of senses offers a successful solution to an environmental challenge, selection may be involved. I have emphasized crossmodality so far because higher-order perception is not possible without the senses cooperating with one another even if they are a team of rivals. So, further discussion of crossmodality and how it affects the biology and behavior and evolutionary potential of organisms becomes an important endeavor in understanding perception.
Combining information from multiple senses is not a uniquely human or even uniquely primate capability. Some mammals combine information from different sensory inputs when the information is perceived to share a link to a larger or overarching sensory item. So, for instance, a high-pitched sound might be coprocessed with light colors because these separate sensory inputs might regularly be associated with a predator or a prey item. There are two possible ways that this can happen, and both might be involved at the same time. An organism may do this, first, because the sensory signals reinforce each other and, second, because they might be processed in similar manners. In the first case, the different sensory signals act like ratchets. When one sense steps up sensory acumen, the other does, too, and so on. In the second case, one sense simply hitchhikes with the other. In addition, it is clear from animal behavior studies that nonprimate mammals can actually formulate very complex perceptions of environmental sensory input, especially when another species is involved (taking us all the way back to “I run away from that,” “I eat that,” and “I mate with that”).
Although little work on this phenomenon in nonprimate mammals has been accomplished at the level of brain structure and physiology, behavioral studies that are akin to the psychological approaches used in human cognitive biology all point to the possibility of crossmodality in many animals. Such crossmodality has an adaptive story behind it because organisms, especially ones that have prominent places in ecosystems, such as mammals, might have a selective advantage if they can rapidly and accurately process and categorize things in their environment. Crossmodality goes a long way to this end. Looking at a few examples will demonstrate the plausibility of this way of thinking about nonhuman primates and other mammals and connections that are made between the senses.
Many of the examples are from carnivores such as cats, dogs, and seals, because these animals are social in nature and observing them has a rich payback. In one example, a naive (meaning that the subject has not previously been tested) captive female sea lion named Rio was given a set of what is called match-to-do tests. The results indicated that Rio could establish crossmodal matching of the visual letters and numbers with sounds.
In match-to-do tests, a behavioral tool called conditioning by reinforcement is used. This approach simply conditions an animal to respond to a particular signal as a result of some reward when the animal gets it right. Two sets of visual classes are established, each with several possible members (fig. 16.1). Rio was conditioned to associate specific sounds with specific items in one class of figures and the same sound with specific items in the other class of figures. She was then trained to classify each of the ten individual figures into their proper classes. So, Rio could associate A with the alphabet class and 4 with the number class. Her conditioning was dependent on the prize she got for getting the classification right and a particular pitch. Rio learned this system quite easily and quickly, and she accurately learned to associate high pitch with letters and low pitch with numbers. She was then introduced to six contrasting tones that were paired (Ring/Siren, Sweep/White, and Pulse/Tone) and different from the original high and low pitch. The six sounds were paired to be like the high-pitch and the low-pitch contrasts. Rio was then “asked” to associate a number or letter with the new sounds. Of the six sounds, Rio consistently associated three of them (Ring, Sweep, and Pulse) with letters and three of them (Siren, White, and Tone) to numbers. Rio apparently established crossmodal contexts for the sounds with their corresponding visual results.
Figure 16.1. The figures on the top line are one class of stimuli (alphabet), and the figures on the bottom are the second class (number).
It has also been suggested that lions, another carnivore species, use crossmodal responses to identify individuals in captivity and in the wild. The researchers who examined this phenomenon used an approach called expectancy violations theory (EVT) for testing African lions in the wild. This approach measures the response of individuals to artificially applied, unanticipated departures from normal social responses. Again, carnivores are a good group to do this with, because they are social animals in general. In this test, lion test subjects heard loud roars from behind a screen where another lion had originally been. The test was designed to induce a reaction from the test lions, indicating recognition of another individual. Lions recognize each other using both sight and sound, but researchers asked, Is the recognition crossmodal? The researchers hypothesized that if no crossmodality was involved in individual recognition, a lion test subject would respond in the same way to a fully visible lion roaring as to a fake, concealed roar. After testing several lions, researchers noted a difference in response: the lions reacted oddly to the incongruent signals and socially normally to the visually connected roars. These results suggest that there is a crossmodal audiovisual connection in individual recognition of these social animals.
Until recently primates have been studied for crossmodal responses because they can be reared at facilities and can be brought into the lab to be tested. This trend is changing dramatically with our recognition of the ethical dimensions of keeping these animals in captivity for research. Nevertheless, macaques and chimpanzees have been the subject of significant research in crossmodality. In a somewhat stunning experiment in 2011, Vera Ludwig, Ikuma Adachi, and Tetsuro Matsuzawa showed that chimpanzees had the same crossmodal connection of vision and auditory senses that humans have. The tests they used challenged chimpanzees in timed manual discrimination response studies. These studies are compelling to watch, because the chimpanzee is placed before a touch-screen computer and stimulated by both sounds and sights and then asked to respond by touching the screen in the proper place. After many tests with several subjects, it was obvious that the chimps associate white with high pitch and black with low pitch. Sound familiar? It should, because it is pretty much the same response that adult and infant humans have to high pitch and light images.
How did this experiment work? The researchers briefly showed a chimp a small white or black square on the computer screen. After a white or black square flashed on the screen, two larger squares (one white and one black) were shown in the upper half of the screen and the chimp was asked to pick the one that flashed before. The researchers recorded how quickly and accurately the chimp responded, and they used these results as a metric in later comparisons with variations of the experiment. The chimps became very used to the test and could do it as quickly as a human. Watching the videos of these tests, one can’t help but root for the chimp to be faster at the task than humans. The experiment began once the chimps got used to the black-or-white identification task. As the squares were flashed, high- or low-pitched tones were sounded as a background noise, and the chimp’s response was recorded for accuracy and speed. If the high pitch was broadcast along with a white square, the chimp more quickly identified the square as white. Conversely, if the high pitch was broadcast with a black square, the chimps slowed down a bit to get the identity of the square correct.
From these experiments one might argue that the crossmodality connecting light colors with high pitch existed in the common ancestor of chimps and humans and perhaps even further back in the primate lineage. Warning! The following may be a Panglossian explanation for how the crossmodal connection of light with high pitch may have evolved, but it’s still fun to think about. (Remember that a Panglossian explanation is one that looks for an adaptive reason in everything.) Some researchers have proposed that the phenomenon of light images being mapped onto high pitch emanates from our lower primate ancestors. The Panglossian argument is based on the notion that there is an adaptive need to process external information fast. The adaptive argument for this connection is also dependent on a trick the visual systems play on us called the light-from-above effect (fig. 16.2). The illustration here shows how the perception of lighting from above drastically effects what we see. The concavity or convexity of the figure is determined by where light is coming from or where we sense that light is coming from. Humans and chimpanzees both assume that light always comes from above—hence, the difference in the perception of convexity and concavity.
Figure 16.2. The light-from-above effect. In two leftmost panels, the visual system processes A as a convex image and B as a concave image. The visual system does this because of the light-from-above effect. To the right, the left two panels have been rotated by 180 degrees. The same figure that was convex (right side up A) is now concave (upside down A) and vice versa.
The Panglossian argument is as follows. We associate light from above with high-pitched sounds because other organisms that we routinely see above us are usually smaller than we are. For example, if you are sitting outside and are wondering what living creatures are above you without looking, you probably don’t even consider that a hyena is up there. Rather, your options are birds, squirrels, or (if you are sitting in Central Park in New York City), the occasional red-tailed hawk. Most of these organisms are indeed smaller than we are and make squeaky, high-pitched sounds. And most of the organisms above us that are squeaky are lit better than organisms that aren’t. The adaptive story has one more step to it. Organisms need to know right away if they need to flee (“I run away from that”) because survival often depends on recognizing something dangerous in fractions of seconds. Usually organisms don’t run away from creatures smaller than themselves and therefore from ones making high-pitched sounds. Conversely, a low-pitched sound means something completely different.
It took me about five hundred words to describe this adaptive just-so story, when a simpler explanation exists. Charles Spence and Ophelia Deroy suggest that the correct correspondences are actually light with size and size with high pitch. Because crossmodal connections are transitive, once the light-size and size-pitch correspondences are learned, this leads to the light-pitch connection. Instead of there being an innate, direct crossmodal connection of light with pitch in the brain, the correspondence might simply be a learned effect. This alternative scenario also suggests that the environmental context of potential crossmodality needs to be examined before we can make claims of innate, direct crossmodality and synesthesia. The Panglossian explanation can trick us much too easily. But if crossmodality is implemented by the wiring of the brain, can we see differences in neural connections in the brain among organisms? And what then makes things different at the neural level as vertebrate lineages diverge?
Some parts of the brain expand, and others commit to more and more connections to other parts of the brain in the evolutionary process. The case of the primate brain needing to be folded into rolls (gyri and sulci) is a good demonstration of different parts of the brain getting larger in our lineage. In fact, it is well known that there was a huge jump in brain volume in the common ancestor of Homo erectus, H. neanderthalensis, and H. sapiens. Chimps and some of our older ancestors related to australopithecines and the very early diverging genus Homo species had brain sizes all less than 43 cubic inches, with chimps at 18 cubic inches. By contrast, the brain of H. neanderthalensis, like that of H. sapiens, was on the order of 92 cubic inches (with the average Neanderthal brain being a little larger than the average H. sapiens brain). Unfortunately, there is no way to delve into what kind of neural connections were in the ancestors we had with these genus Homo species. Unlike Leborgne’s brain, which is preserved in formalin, and Molaison’s brain, which was sectioned into thousands of thin slices soon after his death, we simply do not have preserved brains or images of the brains of these long-extinct Homo species. We do, however, have information on the neural connections of chimpanzees, other primates, and mammals. What can they tell us about the transition into human perception and perhaps even the development of consciousness in our lineage?
The short answer is that only some of the connections are there. The longer answer requires consideration of the connectome. This relatively new brain term refers to the mapping of the axonal connections by means of synapses in the brain and the tendency of modern biologists to append the suffix “-ome” to any approaches that are big data in nature. One way to do this is to simply look at the cellular structure of the brain, as with Henry Molaison’s brain described in Chapter 10. This approach has been used in model organisms with manageable brains such as the nematode (C. elegans). The approach requires that the brain (or prain—see Chapter 2) of this nematode is sliced into hundreds of very thin sections that are then viewed sequentially in a transmission electron microscope. Each section is photographed at a magnification so that every neural cell can be identified. The photographs are then stacked using computer imaging, and the various neural cells can be tracked and plotted. When this is accomplished over the entire brain, the connections of all of the visible nerve cells can be traced. In a tour-de-force study in 1986, researchers were able to determine that the connectome of the nematode consists of 279 cells. That turns out to be about one-fourth of the total number of cells in this tiny wormlike organism of about 1,000 cells. In addition, there are a precise 6,393 synapses connecting these 279 cells, with 1,410 connections of these neural cells to muscles. Yet this very simple worm can sense light, tactile input, and odors, and can taste. There is no evidence, though, that the nematode has any crossmodal sensory capacity.
This approach is as tedious as it sounds but incredibly rich in information. Although it can give cell-to-cell connectome information, other approaches can give a macroscale picture of the connectome, and the methods used to uncover these more macroscale connectomes include MRI and DTI (see Chapter 15). By combining information from hundreds of studies into a rich database, researchers have attempted to construct neural connection networks. The major analytical tool they use to build connectomes of different organisms concerns the use of a branch of mathematics called graph theory. Basically, each picture of the nervous system—whether it is an actual microscopy picture or simply information about connectivity based on other observations—can be represented in a graph that is then used to analyze and interpret the data.
A graph is simply a mathematical description of the network of connections from one region of the brain to another. The regions that are connected are called nodes, and the resulting graph is intensely rich in information. To combine graphs from different sources is not trivial, so graph theory approaches are needed to make sense of the overwhelming amount of data (big data) involved. Graph theory leads to three very interesting aspects of neural networks: hubs, clubs, and communities. A hub refers to a highly connected node or neuron, a club to a collection of neuronal connections that are much higher than the background level of connections, and a community to an interconnected set of neurons. All three aspects of graphs can be used to give a detailed picture of connections in the brains of different organisms and lead to some universals about connectomes.
Graph theory analysis of the connectome shows that, no matter the species, communities of connections are localized to specific parts of the brain. This pattern means that clustering of neurons most commonly occurs in localized, spatially restricted regions in most mammal brains. Some connections reach out from these communities, but the overall preponderance of connections occurs among close or neighboring neural cells. Moreover, the communities can be linked to specific functions, such as the sensory regions of the brain discussed previously (see Chapter 15). There is also a conservation of these regions across relatively closely related species, such that, for instance, mice, monkeys, and humans show some very similar limbic and cognitive networks. If one branches farther out on the tree of life—to insects or even to fish—these similarities either disappear or are very hard to pin down.
And what all of this means is that, as the brain evolved to become larger and larger in the primate lineage, the larger mass resulted in more tightly wired communities of neurons, which in turn led to the potential for specialized functions of the localized communities. Martijn van den Heuvel, Edward Bullmore, and Olaf Sporns suggest in their review of connectomes that the strong modularity of structures that are observed in the connectome have led to the localization of cortical regionalization of sensory and motor information. In addition, they suggest that as this hub, club, and community organization of the connectome emerged, it also led to increased lateralization of the brain and the evolution of specialized neural functions like language. Indeed, the acquisition of higher cognitive functions that make us the species into which we have evolved more than likely would not have occurred without these general rules of clustering of neural cell connections in the connectome. And of relevance to the crossmodal effects and synesthesia introduced in Chapter 15, this general rule about connectomes favored short neural pathways over longer ones as well as more communication between communities that results in crossmodal effects.
Figure 16.3. Connectomes of the human, chimpanzee, and macaque.
But the story isn’t finished, because it doesn’t end with the mere observation of similarities. There are differences, too, between humans and other mammals and especially other primates. The most relevant to the discussion are those among the lower primate macaque; our closest living relative, the chimpanzee; and us (fig. 16.3). The surprising result in comparing the connectomes of these primates with ours is that in many ways they can offer us some hypotheses concerning higher cognitive and behavioral aspects of our evolutionary history.
The connections in the brain that have undergone evolutionary change can be determined using the various brain imaging techniques discussed previously (see Chapter 15). The two major tissues of the brain where connections are critical—gray and white matter—have distinct differences among primates. White matter is white in appearance and made up primarily of nerve cells called axons that are threaded through this part of the brain and act as conduits to the gray matter of the brain. Gray matter is pinkish gray. It is more complex than white matter in that it contains a different kind of nerve cell called a dendrite and also contains the ends of the axons that run in from the white matter. The ends of the axons form synapses with the dendrites. The gray matter is generally found in the outer layers of the brain, and the white matter is situated more toward the inside of the brain.
Connectome tracts are mostly in the white matter, and indeed this is where the most important tracts of neural tissue are involved in more intricate connections. Diffusion tensor imaging is an especially good technique for deciphering the connectivity of tracts in white matter. Humans, in comparison to other primates and even in comparison to other mammals, actually differ in the density of white matter, which means that there are also probably differences in the density of connections. Specifically, humans have a higher ratio of white to gray matter in their brains, and this occurs most conspicuously in the prefrontal cortex (Fig. 16.4).
Figure 16.4. White and gray matter in a cross-section of the brain.
As for connectivity, researchers have determined that only one-fourth of the landmark connections among primates are common to monkeys, apes, and humans. There are also specific areas of human brains that have inordinate numbers of connections, and these correlate with the specialization of function for the region. Good examples of this trend are regions such as Broca’s and Wernicke’s that are involved in such uniquely human capacities as language. Examination of connections using DTI reveals that connections in several parts of the primate brain (mostly areas in the outer middle part of the brain) are conserved from monkeys to apes to humans. But other parts of the prefrontal cortex and the inferior parietal section (a region just posterior to the prefrontal cortex) of the brain show very interesting and more complex divergence in humans.
As lower primates diverged from monkeys and monkeys from the great apes, the brain enlarged and connections in the more derived ape brains (like ours) were added to. The mirror neuron systems in primates are a good example of this trend. Mirror effects were first shown in macaques, and both chimps and humans also show the phenomenon. Brain imaging techniques such as DTI can uncover the neural connections of the mirror system, and researchers have examined the fine detail of such connections. Diffusion tensor imaging has shown that mirror neuron systems have three levels of organization in primates. Macaques, chimps, and humans have mirror neuron connections in the frontal temporal part of the brain, and the connections extend into the frontal parietal region of the brain by means of very specific neural cells. But this region is where the mirror neuron pathway stops in macaques. In humans and chimps, the mirror connections extend into the inferior temporal cortex. But this is where chimpanzee mirror connections stop. In humans alone, the mirror connections extend into the superior parietal cortex. Erin Hecht and her colleagues have constructed a model to explain these differences and suggest that “differences in mirror system connectivity and responsiveness with species differences in behavior, including adaptations for imitation and social learning of tool use,” might have evolved in this way.
But here is another Panglossian moment. It seems that a major goal of the comparison of macaque, chimp, and human brain size and brain connections is to explain the broad difference in behavior of the species involved. There is no doubt that the source of the huge difference we see among ourselves and other primate species is indeed the brain. Yet we must be careful about allowing Dr. Pangloss to make a house call and leap to the conclusion without evidence that these different mirror system brain connections led to an evolutionary jump like tool making. On the other hand, such a suggestion is a wonderful hypothesis for testing.
