If the brain is the physical instantiation of the mind, then every mental event must be reflected in the physical realm, from imagination to hallucination to mental imagery. The current theory in neuroscience is that this physical realm is the firing of neurons—that is, just as we know that the firing of sensory neurons responds to sensory cues, and the firing of motor neurons controls muscles, this theory predicts that nonsensory mental (cognitive) events are not just reflected in but actually are the firing of neurons.
The fact that the cortex can be better understood as a two-dimensional sheet of repeating columns rather than as a three-dimensional structure suggests that we can think of the cortex as a series of interconnected maps. Visual structures map the visual field in front of you, auditory structures map tones and sounds, and both sensory and motor structures map your body.1
In the mammal (including the human), the primary visual cortex happens to be at the back of the brain. The axons connecting your eyes to your visual cortex have to run all the way across the brain (they run underneath), which is why a concussion (in which your brain slams against your skull like an unconstrained object in a suddenly stopping car) often leads to visual problems.2 Recordings of neurons from primary visual cortex find that for the most part, the cells in each visual cortical column are tuned to the same point on the retina. Some of these cells are tuned to spots on the retina, while others are tuned to oriented lines, and others are tuned to moving oriented lines. Some are excited by light at the location, while others are inhibited. But each column in the primary visual cortex is tuned to light arriving at one spot on the retina.3
Connectivity in the cortex is based on a center-surround architecture, in which cells representing similar locations on the retina are more tightly coupled than cells representing distal locations, with a larger-scale inhibitory connectivity. There are also some longer-spaced connections that have subtle effects (they will be particularly important when we discuss migraines and the fortification hallucination), but the primary connectivity seems to be between adjacent columns representing similar information.4 At the simplest level, cortex can be thought of as a sheet of local excitation and broader inhibition. This connectivity produces what is known mathematically as a local coupling kernel, a connectivity repeated at every location.5 Such coupling kernels are seen in physical objects as well, such as the relationship between the coils of a spring or the physical connectivity of a guitar string (in one dimension), the physical connectivity of a bed sheet (in two dimensions), or the relationship between molecules of water (in three dimensions).
Those are three particularly interesting examples, because they all show wave effects. We have all seen waves on water, and we have all snapped a bed sheet flat, sending a wave across it; a classic junior high school physics experiment is to take a spring and send a traveling wave through it. And, of course, the sound generated by a guitar string is based on a standing wave, as it vibrates under the restricted conditions (mathematically termed the boundary conditions) that the two ends are held fixed and cannot move. Just as waves can travel across physical objects, so too can waves of neural activity travel across the cortical sheet.6
What would it feel like to have a wave of activity travel across your visual cortex? If neural activity in visual cortex is visual perception, then you would perceive a wave across your visual cortex as a wave of light traveling across your vision. In other words, you would hallucinate a wave of light. But people have been reporting their hallucinations for thousands of years, and random hallucinations don’t appear as a wave of light across your vision.7
The reason for this is that the primary visual cortex in the primate (including the human) does not represent the retinal input with a one-to-one transformation. Instead, there is more representation of the center of your retina (an area called the fovea) and less representation of the periphery.8 The specific mathematical description for this transformation is called a log-polar transformation. Think of your visual cortex as two rectangles pasted onto the back of your brain, lying next to each other. Each side of your visual cortex represents half your visual field. Cells in the fovea project (through an intermediate thalamic structure that does additional processingA called the lateral geniculate nucleus) to cells in the visual cortex near the midline, while cells in the periphery project (again through intermediate structures) to cells in the visual cortex more laterally. Moving along the horizontal axis of the visual cortex is equivalent to moving out from the fovea. Just as there are more cells in the fovea than the periphery, there is more visual cortex dedicated to the fovea than the periphery. Similarly, moving along the vertical axis is equivalent to moving in a circle around the fovea, with the top of your visual cortex corresponding to the top of your visual field.
This means that a wave that was perfectly aligned to the vertical axis of your visual cortex, one that started at the midline and progressed away from the midline, would appear as an expanding ring. Because more and more visual field is represented by the same-size area of cortex as one moves out to the periphery, a traveling wave moving at a constant speed on the cortical surface would appear to speed up as it travels to the periphery.
Simple visual hallucinations are often seen when falling asleep and waking up, under certain flickering lights, and in sealed dark rooms. They are also seen under the influence of certain recreational chemicals (drugs). These simple visual hallucinations are very similar to those painted on and carved into sacred stones and cave walls. People have been reporting their chemically-induced hallucinations for thousands of years. They are also the same patterns often seen in “near-death” experiences.9
These images tend to be made of radial lines, spirals, and spider-web shapes. They often appear as spiral checkerboards, honeycombs, or dots. And sometimes they are as simple as a white light in the center of the vision. Sometimes the images are stable and sometimes they move in a spiral inward or outward pattern. The images move with the retina (and are thus seen in “retinal space,” which is particularly strange to experience because we are used to seeing things in “allocentric” or real-world space, which moves on our retina when we move our eyes). These images are the visual interpretation of waves in the cortex.
A vertical wave flowing from top to bottom along the visual cortex will appear as rotating radial lines. A collapsing wave flowing from the lateral to medial aspects will appear as a collapsing light, and may easily be interpreted as a tunnel. A wave that is diagonally oriented along the visual cortex will appear as a spiral. As the wave moves, the spiral can appear to spiral inward or outward. See Figure 17.1.
The primary visual cortex in the primate is actually connected up in a more complex manner than a simple center-surround kernel.10 The cortical columns are arranged into local hypercolumns, which represent a single location on the visual field, but contain orientation-selective cells, which respond to lines oriented in different ways. A hypercolumn contains columns of cells representing each potential orientation. Each column is coupled to cells within its own hypercolumn (because they represent the same location in the visual field) and to nearby columns in other hypercolumns that share its preferred orientation (because they represent lines of the same orientation). The cross-hypercolumn connections seem to be laid out in an approximately hexagonal grid. It is the hexagonal grid across hypercolumns that produces the checkerboard or honeycomb components of some visual hallucinations.
The visual system in the primate (including the human) is very complex, including a host of separate cortical areas, each of which is dedicated to a different useful feature, such as colors, spatial locations in the visual world, even faces, rooms, and other dedicated objects.11 Current theories suggest that this complex network of visual fields is a set of separate, dedicated feature detectors, tuned to the set of natural images we experience in our lives.
Hallucinations of more complex events and objects are likely to depend on the firing of other cortices, such as those representing objects, faces, locations, and other sensations. However, because we don’t know how these higher-order representations are encoded, we don’t know how waves would appear on them. Random firing in the visual cortex is also seen under hallucination conditions, which appear as small colored spots on the visual field (again, that follow your retina as it moves).12 Just as the firing of the primary visual cortex is perceived as vision, the random firing of neurons in the face-representation area (a small, dedicated piece of the cortex called the superior temporal sulcus or STS)13 would be perceived as seeing a specific person.
Figure 17.1 The log-polar transformation in the primate visual cortex. Patterns on the visual cortex (horizontal lines, vertical lines, diagonal lines) are transformed into retinal perceptions (radial lines, concentric circles, a spiral) because of the log-polar transformation from the retina to the visual cortex. Similar patterns are seen during sensory deprivation, drug-induced hallucinations, and near-death experiences. Similar patterns also appear drawn on ancient petroglyphs.
One of the best-described visual hallucinations occurs during migraines and is described as the “fortification” hallucination or “visual aura.”14 Generally, these are experienced as extremely bright jagged edges, progressing slowly across the visual field, followed behind by a visual scotoma or blackness, obscuring one’s vision. Interestingly, the line of jagged edges speeds up as it progresses from the center of one’s vision to the edge, over the course of 10 to 20 minutes. At this point, you may be able to guess why the jagged edge speeds up as it progresses out from the center to the periphery of vision.
These “fortification” auras are directly explainable as the firing of cells in the visual cortex.15 As noted above, cells in the visual cortex represent visual information by being tuned to short, oriented lines. Different orientations within a hypercolumn are connected to each other. Thus, if much of a hypercolumn was active, it would appear as a set of angular lines, at sharp orientations to each other. Because the columns are connected across hypercolumns, one sees a continuous hexagonal wall. As the wave progresses from the medial edge (the center of one’s vision) to the lateral edge (the periphery of one’s vision), the wall seems to move. It is exceptionally bright because all of the cells are firing as the wave progresses through them. Firing uses up cellular resources, and cells become unable to fire (become refractory) afterwards.16 The inability of cells to fire behind the wave appears as a blank darkness or a scotoma. The wave appears to speed up because our cortex represents the center of our visual field with more cortical area than the periphery. Thus a constant-speed wave in the visual cortex covers more and more of our visual field as it progresses laterally and appears to speed up as it moves.
The term “fortification” is an allusion to St. Hildegard von Bingen, a 12th-century nun, living in Germany, who left descriptions of visions of lights more brilliant than the sun progressing through her vision, which turn into black coals, of visions of a spreading fire of God, followed by gloomy darkness, leaving ethereal stars behind, and of visions of a head of fire scourging the darkness.17 Several authors in the 20th century have suggested that St. Hildegard’s visions were migraine auras.18 The jagged edges of the fortification illusion do, in fact, look like crenellations and battlements that might be seen atop a medieval castle. They are often brilliantly bright (being the firing of many cells simultaneously) and followed by a darkness (as the cells shut off and become refractory, tiring out and becoming unable to fire), which obscures vision (because vision is the firing of specific cells; which subset of cells fires is the information about what you see—if all the cells are silent, one sees only darkness). Figure 17.2 shows a modern depiction of the fortification illusion. Hildegard explicitly described that the visions occurred during normal wakefulness, not during sleep or trances, and that she could not see the length or breadth of these visions (presumably because the images were on her visual cortex, so they moved with her vision when she moved her eyes). One can easily imagine the effect on a young woman in the 12th century of an inexplicable jagged bright line leading an area of darkness expanding through her vision, especially one that continued whether her eyes were open or closed, that she could not look away from, and that was accompanied by a blazing, inescapable headache.
Figure 17.2 Fortification illusions in migraines. (a) Migraine sufferers see scintillating zigzags followed by darkness. A migraine sufferer watched a flickering display and drew what was seen. (b) The subject was asked to focus at a single point and draw the edges of these fortifications on a piece of paper at 2-minute intervals. The results reveal that the fortifications expand outward at accelerating speeds. From Grüsser (1995), reprinted with permission from the publisher.
If waves in the cortex and the random firing of cells produce hallucinations and the illusion of seeing things that are not there, what about the other direction? When we imagine something, is that also a physical process? This has been most thoroughly studied through visual imagery, in which people are asked to imagine a visual image.19
The first study to directly address this question was an examination of the ability to mentally rotate images in one’s mind. In a now-classic experiment, Roger Shepherd and Jacqueline Metzler asked subjects to determine whether two pictures of three-dimensional objects were pictures of the same object rotated through space, or whether they were different objects.20 What they found was that the time it took a person to determine if the two objects were related was a linear function of the rotational angle between the two objects. They concluded from this that people were mentally rotating the objects (at a rate of about 60 degrees per second). Of course, when we look at the two objects and try to compare them, we have the subjective feeling that we are mentally rotating them (at least I do!). These rotations appear to activate visual and motor structures, as if people were imagining rotating the objects manually.
Following up on this work, Stephen Kosslyn and his colleagues began testing the psychophysicsB of mental imagery.21 If I ask you to imagine a pirate boat and then to talk about the prow of the boat (What is the figurehead on the prow of the boat?), you are faster to answer questions about the prow than about the stern (How many guns are pointing aft?). Somehow distances matter in the mental imagery. These visual representations have been validated by fMRI results that have shown that mental imagery activates the appropriate sensory areas of the brain: visual mental imagery entails activity in the visual cortices and auditory mental imagery entails activity in auditory cortices, etc.
Of course, there is more to the brain than the primary sensory cortices. Normal people know the difference between imagining a boat and seeing one. This means that somewhere in the brain (likely the frontal cortices), other neurons are firing that differentiate them. One theory about hallucinations is that hallucinations entail mental imagery without those other neurons saying, “this is just your imagination—it’s not real.”22
A particularly interesting example of this sort of mental imagery has captured the imagination of neuroscience and psychology recently—motor control neurons in the monkey premotor cortices (the motor-related planning cortices23) show similar activity whether the animal is performing the movement itself or whether it is watching another primate perform the movement.24 These neurons have been called “mirror” neurons; however, it is not clear at this point whether they are a special type of representation in the primate cortex or whether they are simply reflecting the animal’s imagination of the other’s actions in its own motor-control neurons.25 We have encountered these mirror neurons previously in Chapter 14, where we discussed them in terms of our ability to empathize and sympathize with others.
Mental imagery is the firing of neurons in the brain. This means that we might be able to identify a covert cognitive process if we recorded enough neurons and identified manipulations of a covert variable.26
The first study to do this was done in the 1980s by Apostolos Georgopoulos and his colleagues, who trained a monkey to reach toward a target.27 Initially, the monkeys were trained to touch a center light and hold it until the light went off, at which time one of eight surrounding lights would turn on. The monkey was simply to reach to one of the surrounding lights for a juice reward. What Georgopoulos and his colleagues found was that neurons in the primary motor cortex were tuned to the direction the animal was reaching and that from a large set of neurons (thousandsC), they could decode the direction the animal was reaching (or was about to reach).29 They then trained the monkey to reach not to the lit-up light on the outer circle, but rather 90 degrees counterclockwise to it. When they decoded the neurons, they found that the ensemble originally decoded to represent the direction of the lit-up light and rotated (through the intermediate directions) to the direction the animal was going to reach before the animal actually began its movement.30 The animal was mentally rotating the reaching direction from the target to the cue.D In fact, the speed of that mental rotation in the monkey motor cortex matched the speed of mental rotation found by the classic Shepherd and Metzler study in humans from two decades earlier in which humans matched images of three-dimensional blocks.
If it is possible to decode covert cognitive and imagined information from neural ensembles, then it should be possible to record the neural activity from the brain, to ask subjects to imagine moving their arms, to decode the neural activity of how they want their arms to move, and to move robotic arms to match. This is the promise of brain–machine interfaces and sounds like science fiction, but it is already a reality in limited conditions.32
Using the same motor-tuned areas that Apostolos Georgopoulos recorded from two decades earlier, Andy Schwartz (then at Arizona State University, now at the University of Pittsburgh) trained a monkey to track a ball in a virtual-reality three-dimensional space.33 Wearing specially designed goggles, the monkey saw two balls—a yellow ball that appeared at a random point in three-dimensional space, and a blue ball that it controlled. If it could get the blue ball to catch the yellow one, it would get a juice reward. The secret (for which the monkey was never explicitly trained) was that the movement of the blue ball was not based on the movement of the monkey’s hand but instead on the decoded information from the monkey’s brain. When he presented his data, Andy Schwartz used to show a video in which the monkey initially tracked the virtual ball with its hands, but then as it realized that the tracking was moving faster than its hands, it dropped its hands to its sides and continued tracking the ball using only its mind. For obvious reasons, this video was affectionately referred to in the neuroscience community as the “Jedi Monkey video.”
More recently, it has become possible to decode this imagined information to move robotic arms, not just in monkeys but now also in humans.34 The work on this has been slow because it required two separate breakthroughs to work—from neuroscience, it required decoding algorithms that could identify what the arm was supposed to do, and from robotics, it required reliable robot arms that could be controlled with sufficient accuracy and simplicity. But both of those breakthroughs have been accomplished and several labs have now driven robotic arms from mental imagery. In 2008, a monkey in Andy Schwartz’s lab was able to feed itself using a robotic arm.35 John Donoghue’s lab has shown that tetraplegic patients (who are paralyzed from the neck down) can guide robotic hands to move and to open and close.36 In one of the videos that John Donoghue shows at his presentations, one of the patients, watching the robotic arm grasp a soda can and lift it up under the control of his own mental imagination, stares at the robotic hand, moving with his thoughts, and says, in clear amazement, “Whoa.”
The concept of a separation between brain and mind (dualism) can still survive the observation that mental imagery is reflected in neuronal firing, because it could be that the brain only reflects what is going on in the mind. The real test of whether the mind truly corresponds to neuronal firing in the brain is whether manipulations of that firing change experiences. In fact, at the sensory level, this technology is so successful that we often don’t realize that it exists. The cochlear implant, a technology implanted in hundreds of thousands of patients, stimulates sensory neurons in the ear to translate auditory signals from a microphone directly into the brain.37 Retinal stimulators translating video images into sensory signals on the retina are currently in Phase II clinical trials.38
However, clinical stimulation of the deep brain beyond sensory cortices has been limited to simple waveforms and frequencies without information. The main clinical stimulation of nonsensory structures has been a procedure called deep-brain stimulation, which is currently a common treatment for Parkinson’s disease.39 Although the mechanism by which it works is still unknown, stimulating the basal ganglia at low frequencies (under a few hundred cycles per second) often re-enables motion in akinetic patients (patients who can’t move). Some researchers think that this stimulation effectively shuts off the basal ganglia structures, producing a lesion-like effect, allowing other decision-making systems to take over. Other researchers think that the basal ganglia need to work at a certain rhythm,E and the stimulation enables it to get back into that rhythm. Deep-brain stimulation targeted at other areas is now being tried for other disorders41 (such as epilepsy and depression), but most of these experiments are pretty much flying blind—no one knows why they are working when they do or why they don’t when they don’t.
Research-wise, however, stimulation has been shown to be able to control everything, including pleasure, pain, and memories. In Chapter 4, we encountered medial forebrain bundle stimulation, where some stimulation targets produced euphoric pleasure and other stimulation sites did not. Stimulation has been used for everything from pain management to muscular control.42 The motor cortices themselves were defined in the 1800s when scientists passed small amounts of current and stimulated the cortex, producing motion in anesthetized animals.43 Stimulating one location on the motor cortex led to arm motion, while stimulation of another led to leg motion.
In the 1940s, Wilder Penfield and Herbert Jasper were trying to help epilepsy patients avoid just the sort of disaster that led to H.M.’s memory loss, and examined the effect of cortical stimulation in these epilepsy patients. Using local anesthesia to open up a small piece of the skull,F Penfield and Jasper stimulated areas of the cortex of awake, resting patients,44 and found that stimulations could trigger memories, hints of songs, even smells, and visions.
The most impressive manipulation of brain signals is still the work by Bill Newsome and colleagues from the 1990s, in which they were able to change a monkey’s perception enough to change its ultimate decision.45 Newsome and colleagues trained monkeys to observe the flow of random dots on a screen. Some of the dots moved truly randomly and some moved in a given direction (say left or right). The monkey’s goal was to determine which direction the (very small) proportion of dots was moving. We encountered this task earlier in our discussion of integrating perceptual signals (Chapter 11). If you remember that discussion, one of the visual areas (MST) determines whether the whole field is moving together. Because cortical representations in MST are arranged in columns, with each column representing a different direction, Newsome and his students were able to stimulate single columns in MST and change the perceived direction of motion, and the monkey’s responses. Similar experiments have been able to change the direction of visual saccades (by stimulating other areas, as well as in subcortical structures such as the superior colliculus, which drives orientation of attention).46
The brain is the physical instantiation of the mind. Interfering with brain function changes mental function. One of the reasons people are worried about learning how the brain works and learning how the physical brain instantiates the mind is that they are worried that we will develop a form of mental control, where machines manipulating your brain control your thoughts, desires, and actions. But just as changing the brain changes the mind, changing the mind changes the brain. Whenever you learn something new, your brain is physically changed. When you realize that you really do have two choices—to drink or not to drink—that is a physical change in your brain as well as a change in your mental state. Humans are already very adept at controlling each other—a tremendous amount of our brain power is dedicated to predicting what our neighbors, colleagues, friends, and enemies will do. It is said that charm is the ability to get people to do something that you want them to do without them realizing that you got them to do it. Just because the manipulation is not physical does not mean it is not manipulation. The brain is the physical instantiation of the mind. Every mental event is reflected in the physical realm, from imagination to hallucination to mental imagery.
• Paul C. Bressloff, Jack D. Cowan, Martin Golubitsky, Peter J. Thomas, and Matthew C. Wiener (2002). What geometric visual hallucinations tell us about the visual cortex. Neural Computation, 14, 473–491.
• Stephen M. Kosslyn.(1994). Image and Brain. Cambridge, MA: MIT Press.
• Adam Johnson, André A. Fenton, Cliff Kentros, and A. David Redish (2009). Looking for cognition in the structure in the noise. Trends in Cognitive Sciences, 13, 55–64.
• Andrew B. Schwartz (2007). Useful signals from motor cortex. Journal of Physiology, 579, 581–601.