The mind is physically instantiated by the brain, but the mind is not simply software running on the hardware of the brain. The physical structure of the brain changes the computations that are available. We saw earlier that there are multiple action-selection systems, each with different advantages and disadvantages. Although it is possible to learn new tasks after brain lesions, it is not that the same software is running on another part of the brain; instead, other action-selection systems are doing their best to accomplish tasks.
Historically, debates about the relationship between mind and brain have been between those who argued that they were fundamentally different things (dualism) and those who argued that the mind was instantiated in the brain.1 There is strong evidence against dualism because manipulations of the brain affect the mind. There is no evidence for a nonphysical entity, and few scientists still consider it a possibility. However, more recently, dualism debates have been between whether the mind depends on the physical brain, or whether it is a software that just happens to run on neural hardware.2
We now have examples of things that can be physically translated from one instantiation to another without any loss. When we download a song from the Internet, the music has been translated from sound waves in the air through a microphone into a digital representation on a computer hard drive, translated from that onto a series of electrical pulses over a wire, often through an intermediate step of light pulses over a fiber-optic cable, then back into electrical pulses, back into a digital representation on your local hard drive, and then back into sound waves in the air through speakers attached to your computer. Although there is some change from the sound waves to the digital representation and back, there is no change at all from one digital representation to another. The physical nature of the mind is not like that. To understand how the material nature of how the brain processes information is different from the immaterial nature of digital information, let us take a moment to understand just what digital information is.
Within a computer, information can be transferred from one place in memory to another (as when you copy a file from disk to memory) or, within a network, from one computer to another. The information may be stored in physically different mechanisms, but the information itself is the same. For example, digital information stored in the form of magnetic orientations on a hard drive can be written to a compact disc (CD) in the form of dark spots on an optical disk.A This has led computer scientists to separate the concept of hardware (the physical entity) from the concept of software (the digital information).
There are cases in biology where information is stored digitally. For example, DNA stores the genetic sequence as a series of four amino acids (guanine, cytosine, thymine, and adenine, usually labeled GCTA). One can make a case that the information in DNA is software and the cell’s complex mechanisms that turn that information into proteins and other physical changes in the cell are the hardware that reads and operates on the software.B
In the 1970s and 1980s, the hardware/software analogy became popular, both among scientists and the public, as an explanation for the mind/brain problem, with the idea that the brain was the hardware and the mind was the software.4 This software/hardware dualism was particularly appealing because it meant that the mind occurred at a different level than the brain and could be studied separately. It also held out the hope that a computer program could be built using a completely different hardware and still create an artificial intelligence.
At the time, the software being studied was generally symbolic—sets of arbitrary symbols that were being manipulated through specific algorithms.5 In 1956, Allen Newell and Herb Simon showed that a search algorithm that looked through possibilities of mathematical steps could find basic mathematical proofs, showing that they could replicate many of the proofs from Bertrand Russell’s Principia Mathematica.6 In fact, their program found a more elegant proof to one of the theorems in Russell’s Principia than Russell had.7
With faster and larger computers, the search process that was developed by these early artificial intelligence researchers has been able to do remarkable things8—it is as good as or better than human doctors at identifying diseases,C it is used by oil-exploration industries to find oil pockets deep in the ground, and, famously, it beat the best chess player in the world. Recently, the combination of search processes with a deep knowledge base of semantic relationships beat the best human players at the TV game show Jeopardy. However, it is important to remember that this search-through-possibilities theory is a hypothesis as to how humans make decisions. Extensive psychological tests have shown that humans do not search as many steps into the future as these programs do.10 Better chess players do not look farther ahead than weaker chess players. Doctors have to be explicitly trained to use differential diagnosis paradigms, and even well-trained doctors tend to use experience-based pattern recognition rather than explicit reasoning in practice.
The search-through-possibilities theory requires a manipulation of symbols. The reason artificial intelligence researchers started with symbol manipulation (as told by Allen Newell in one of his last public lectures given at Carnegie Mellon11) was that scientists thought that “perception was going to be easy,” so they would start with the hard part of cognition. At the end of his career, Allen Newell came to the conclusion that in fact they had been going at it backwards, that perception was far from easy and that cognition was actually perceptions applied to the perception mechanisms. But at the time (from the 1950s through the 1980s), artificial intelligence entailed manipulations of symbols and led to what Newell and Simon called “the physical symbol systems hypothesis,” which proposed that cognition consisted entirely of the manipulation of abstract symbols.12
Even in the 1980s, the concept that the mind entailed the manipulation of symbols was attacked, primarily by John Searle, a philosopher at the University of California, Berkeley. Searle attacked the symbol-manipulation hypothesis with a thought experiment that he called the Chinese room.13 To understand Searle’s thought experiment, we need to first start with the famous Turing test, proposed by Alan Turing in 1950.14
How do I know that you are conscious? Turing rephrased this question in terms of thinking, as in Can a machine think? As we have seen throughout this book, in order to be scientific about this, we need to operationalize this question. Turing suggested that we can operationalize the concept of thinking by trying to determine if the person you are having a conversation with is a computer or a human. Unfortunately, humans are actually very bad at this and often attribute human characteristics to nonhuman objects (such as naming a GPS).
Searle’s Chinese room thought experiment is a direct attack on the concept of the Turing test. In Searle’s thought experiment, imagine that a Chinese speaker is having an email chat with someone in Chinese.D Our speaker types Chinese ideograms into the computer and reads Chinese ideograms on the computer. Somewhere else, a non-Chinese speaker sits in an enclosed room with a set of index cards. Chinese characters appear on the screen in front of him, and he matches the ideograms (which he does not recognize, but we assume he can match them pictorially) to index cards that have complex if-then statements on them. Some of those cards tell him to draw specific ideograms on a tablet, which is then translated to the outside.E Searle correctly points out that even if the outside Chinese speaker (our user) cannot tell the difference between the enclosed room and another Chinese speaker (that is, the room passes the Turing test), we cannot say that our inside man speaks Chinese. This is completely correct. However, it also completely misses the point of the Turing test. While our inside man does not speak Chinese, the room (the man manipulating the index cards, the rules on the index cards themselves, and the input–output system) does.
This is a common mistake made by people trying to understand brain functionality. We keep looking for mechanisms where consciousness resides (in a specific neuron subtype or in the quantum fluctuations of microtubules16), but this is not how the brain works. The brain is an interactive, computational machine that takes information in from the world, processes that information, and then takes actions to respond to it. There are many modules in the brain (we explored some of them in the previous section of this book), but, in the end, it is the person as a whole that takes an action. When you hold your sleeping baby daughter for the first time, that uncontrollable, overwhelming emotion created by your Pavlovian action-selection system is you, not someone else; when your Procedural system slams on the brakes of your car and avoids the accident, that was you making the right choice. Conscious or not, it’s still you.
There are two important hypotheses underlying the hardware/software distinction. The first hypothesis is that there are different levels of description of an object.17 One can describe the digital information stored on a hard disk or DVD in terms of the physical object that it is stored on (for example, as changes in orientation of the ferromagnetic domains on the hard disk) or in terms of the information that is stored (as the sequence of zeros and ones). The second hypothesis is that the information level is independent of the physical description18 (that is, the sequence of zeros and ones stored on the hard disk is the same as the sequence of zeros and ones stored on the DVD). Both of these statements are incorrect when applied to the physical brain. While we will find both the physical and computational or informational levels useful in understanding brain function, experimental evidence suggests that the hypotheses that these are distinct and separable descriptions does not apply to the brain.F
The hardware/software analogy arose from two scientific literatures (an inability to explain consciousness from physical effects20 and observations of patients recovering from lesion studies21) and gained traction as artificial intelligence researchers tried to show that symbol manipulation could explain intelligence.22 Psychology experiments have found that symbol manipulation is particularly hard for humans, and that the more expertise one has with a subject, the less symbol manipulation one seems to do.23 This suggests that there is something else going on beyond symbol manipulation.
It is tempting to throw up our hands and say that because we can’t yet explain how consciousness arises from the brain, it can’t be a physical object. However, physical manipulations affect conscious thought,24 and operationalizing consciousness has been elusive.25 Every time scientists operationalize a part of consciousness so that it can be studied experimentally (for example, by concentrating on one aspect such as attention, perception, emotion, or voluntary motor control), these studies have produced detailed physical (and computational) explanations for how the process works in brain tissue.26 Similarly, when we actually test the timing of consciousness, much of it seems to occur after rather than before behavior.27 Not having an explanation for consciousness seems to be a rather weak starting point for the major theory that consciousness is the software running on the brain’s hardware.
Historically, however, what really seemed to drive the software analogy was the observations that animals and patients could sometimes recover from brain injury.28 Patients with aphasia (language impairments) due to damage to part of their brain would sometimes (with sufficient training) recover some language abilities.29 Animals with brain lesions would sometimes be able to learn to do tasks that they couldn’t do shortly after the lesion.30 And, most importantly, long-term memories seemed to be stored in a different part of the brain from recently stored memories.31
The idea that the brain is an amorphous container and that all parts of the brain are equivalent goes back to a famous experiment by Karl Lashley in the 1920s in which he observed rats running mazes with large swaths of brain tissue removed.32 Lashley certainly recognized that there was a difference between parts of cortex (for example, visual cortex was obviously specialized for vision, even in the rat33), but Lashley found that the ability of rats to solve a maze depended on the amount of brain tissue removed—the more tissue he removed, the worse the animal did. Generally, Lashley didn’t see much difference between lesions that were of similar size but that involved different parts of the cortex—but he also wasn’t watching the rats do much in the first place. When we actually observe not just whether an animal accomplishes a task, but how the animal accomplishes the task, what we find is that animals (including humans) bypass brain lesions not by moving information but by solving tasks using other systems.
We have already seen a difference between Deliberative decision-making (Chapter 9) and Procedural decision-making (Chapter 10). This distinction matches very closely a distinction seen in the human psychological literature between declarative and procedural memory.34 Although the idea that there were multiple, dissociable memory systems dates back to the 1970s, the specific distinction between declarative and procedural memory in human learning, and the specific idea that they were dissociable, goes back to a 1980 paper by Neal Cohen and Larry Squire and the famous patient H.M.35
H.M. (now known to be Henry MolaisonG) was a young man who suffered terribly from intractable, debilitating epileptic seizures that made it impossible for him to work.37 Epilepsy is a breakdown in the negative feedback systems in the brain that normally prevent the highly interconnected positive (excitatory) connections between neurons from running amok. Like an avalanche or a wildfire, one over-excited event can spread throughout the brain.H
In 1953, at the age of 27, H.M. agreed to a highly experimental (and presumably desperate) procedure being done by William Scoville at Hartford Hospital in Connecticut: they would remove the part of the brain that contained the epileptic focus, the starting point of the epileptic avalanche.42 In the subsequent procedure, Scoville removed both of H.M.’s hippocampi and much of the surrounding tissue (called the “medial temporal lobe”).
This procedure was introduced by Wilder Penfield and Herbert Jasper at the Montreal Neurological Institute in the early 1940s.43 It is still done today as a last resort for intractable epilepsy.44 But today, before any tissue is removed, doctors spend time locating the focal site (the starting point) of the epilepsy by recording for long periods of time (even up to weeks) to make sure that they localize the focus as tightly as possible, and then they spend time determining, as best they can, what that part of the brain does. In 1957, none of these techniques were available; all Scoville and his colleagues had available was the electroencephalogram (EEG), which doesn’t have enough resolution to break down the information being processed, and which was only good enough to show him that the epileptic focus was somewhere in the medial temporal region. So, desperate to stop the epileptic seizures, H.M. agreed to let the surgeon remove a large portion of his hippocampi and his medial temporal lobe bilaterally.
The brain (and the mind) are physical things—removing a part of the brain removes a part of the computation. (Sometimes other parts can step in to accomplish a task in a different way, but sometimes not. This is the fundamental error in the software/hardware analogy.) At the time, in 1957, the computational role of hippocampus was still unknown.
In terms of its immediate effect, the surgery improved H.M.’s epilepsy dramatically and made it possible to control the epilepsy with medications. His general intelligence did not seem to be affected and his linguistic abilities remained normal, but the surgery left H.M. with a devastating memory impairment—he couldn’t remember anything new. While he could remember his past, new information could be remembered only as long as he attended to it.45 If he was distracted, it was gone. This strange memory disability had been seen before and was known as anterograde amnesia.46 It was often seen in patients with Korsakoff’s syndrome, a disease caused by chronic alcoholism combined with a vitamin B (thiamine) deficiency.47 Case studies of anterograde amnesia are described beautifully in Oliver Sacks’ The Man who Mistook his Wife for a Hat (see “The Lost Mariner” and “A Matter of Identity”), in A. R. Luria’s The Man with a Shattered World, and (fictionalized into a complex mystery story) in Christopher Nolan’s Memento.
H.M. was a patient who was normal (except for his epilepsy), of above-average intelligence, with a sudden loss of this very specific ability to learn new things. Here was a chance, in this terrible tragedy, to learn what went wrong and to understand how not to make this same mistake in the future.48 Scoville called in Brenda Milner, a neuropsychologist who had studied with Donald Hebb and Wilder Penfield at McGill University. While Milner found that H.M.’s inability to learn new memories was pervasive and crossed all subjects and domains (visual, auditory, linguistic, etc.), she also found that H.M. could learn a mirror-tracing task that took both H.M. and control subjects several days to learn.49 Along with her student Suzanne Corkin, they found that H.M. could, in fact, learn new tasks, but wouldn’t remember doing them. These tasks were all the kind that take lots of practice and are learned slowly, like mirror-writing or tracing intricate paths. Corkin and Milner described how H.M. would be incredulous that he could do a task because he could not remember that he had ever tried it before, yet the rate at which he learned these slowly learning, practice-requiring tasks was approximately normal.
This led to the suggestion that there were two memory systems—one that learned stateable facts (“My car is parked at the airport.”) and another that learned skills (riding a bicycle, throwing a baseball). Facts could be declared—I can tell you where my car is (“at the airport”) and you know it immediately. But skills are procedures that can only be learned through practice. No matter how much I tell you how to throw a baseball, you won’t be able to do it until you try it for a while. The first kind of memory was termed declarative, the second kind procedural.50
Subsequent work has suggested that declarative memory has two components, an episodic component including memories of specific places and times (the time I forgot my wife’s birthday) and a semantic component consisting of facts (the actual date of my wife’s birthday).51 We now know that episodic memory is a constructed imagination of the past—it likely corresponds to the same system used to imagine the future.52 This is the Deliberative system that we described earlier (Chapter 9). Semantic memory does not yet have an accepted correspondence in our decision-making taxonomy but may be similar to the situation-recognition and narrative components (Chapter 12).
What is important here, and what dooms the hardware/software analogy, is that patients such as H.M. are not shifting the software from one part of the computer to another. Rather, each component is able to perform certain calculations on the incoming information. When the lost mariner in Oliver Sacks’ case study learns to become familiar with his life among the nuns and finds peace in his garden, he has not found a way to regain his ability to learn new memories. Instead, he has found peace in the way he uses other systems to accomplish his tasks. When H.M. learned to accomplish mirror-writing, he learned it using the procedural memory system.
H.M. lost his hippocampi and other specialized subcortical structures.53 Subcortical and cortical processing are quite different: while subcortical structures tend to be specialized, the cortex is a large two-dimensional sheet of repeated processing units, arranged in small, repeated columns.54
Current theories suggest that each repeated part of the cortex performs a similar computation on a unique set of inputs. (Exactly what that computation is remains controversial, but it seems to be some form of categorization process occurring via a content-addressable memory mechanism.55 See Appendix C for a discussion of what is known about the computations occurring in the cortex.) However, that computation is fundamentally dependent on the inputs, and, unlike digital computers, those inputs are incomplete.56 That is, you can’t send any input anywhere you want in the cortex. Each cortical column receives a limited set of inputs, and the information carried by those inputs is all that it has available.
We can see the limitations of this by looking at how cortical representations can shift with changes in sensory inputs. This has been studied in the most detail in the context of primary sensory systems (particularly somatosensory [touch] systems).57 Cells in these sensory areas receive inputs from a broad range of input cells, with stronger inputs usually at the center and input strength falling off as the input changes.58 We talk of “tuning curves,” which generally have a peak sensitivity but a broad range (see Appendix B). The cells compete with each other to determine which signals will form the peak of each tuning curve. Take your palm, for example: at each point on your palm, a host of mechanoreceptors have molecular mechanisms that detect touch. These detectors lead the cell to fire spikes, which are transmitted to the spinal cord and then through multiple steps to your sensory cortices, where they are interpreted as touch sensations. Even though these individual mechanoreceptors cover very large areas of your palm, you can localize a pinprick touch to a small point because your somatosensory cortex implements a sort of winner-take-all process using a mechanism called lateral inhibition, in which cortical cells inhibit their neighbors. This means that although lots of cells received input, only the ones representing the touched location remain active. This system integrates information from a very large area but provides a very accurate interpretation of the sensation.
More areas of your somatosensory cortex are sensitive to your palm and fingertips than to your arm, which makes your palm and fingers a more sensitive sensor than your arm. A classic high school experiment is to close your eyes and have a lab partner touch two pencils to your hand or arm or back. Can you tell that they are two points, or do you feel it as one? (As a control, the experiment is usually done with the lab partner sometimes using one pencil and sometimes using two and then asking the subject “one or two?”) The sensitivity of your palm is much more accurate than your arm or back. This nonuniform distribution occurs in all sensory systems. Your auditory cortex spends more cortical area interpreting the frequencies at which human speech occurs than other frequencies, while your visual cortex spends more cortical area interpreting the center of your vision (called the fovea) than the periphery.59
But what happens when one loses part of the input? Because the cells are actually receiving input from large areas of the sensory field, but the far inputs are usually competed out, if the central part of the input is lost, the far inputs will now win the competition, and the cell will shift the area it listens to and show responses to the areas that remain.60 Notice that although the compensation is a change in what information is processed by the cortex, the compensation is not due to a shift in software to the no-longer-used cortex; rather, it is due to a shift in which already-wired-up inputs are being listened to.
This effect produces interesting consequences, particularly in the sensations in phantom limbs. Patients who have lost a limb often feel sensations in that vanished limb. V. S. Ramachandran has shown that these sensations are due to sensory stimuli arriving at structures represented nearby on the somatosensory cortex.61 The layout of the primate somatosensory cortex has been known since the late 1800s.62 It has some strange discontinuities—for example, the fingertips are represented near the face. Patients missing a hand will sometimes feel sensation on their hand when their faces are touched. Patients with amputations often complain of pain or itching in their phantom limb; these sensations are extremely uncomfortable because they cannot scratch a limb that isn’t there. But scratching the adjacently represented area can sometimes relieve the phantom limb pain.
If the inhibition between areas is incomplete or disinhibited or the connections are incompletely pruned or overconnected during development, these connections can cross between sensory systems and lead to experiences of relationships between sensations—one can hear colors or see sounds.63 This process, called synesthesia (from syn, meaning “with” or “together,” and asthesia, meaning “feel” or “perceive” [as in anesthesia or aesthetics]), is a fascinating interaction between sensory systems. Intriguingly, these relationships are not random; they tend to be constant within a given person. Flavors taste like shapes. Numbers have associated colors.64
There is some evidence that during development, cortical representations can shift more than during adulthood.65 In part, this seems to be because cortical connectivity in juveniles and children is broader and gets pared down as cortical systems develop. This has been studied most quantitatively by Eric Knudsen, looking at how representations in the colliculi of owls change in response to sensory changes. The superior and inferior colliculi are subcortical brain structures in birds and mammals that perform an attention or targeting function—the cells are organized in a topographic manner representing position around the head of the animal, horizontally and vertically.66 Stimulation leads to the animal attending to that horizontal and vertical position. The cells in the inferior colliculus respond primarily to auditory (but also more weakly to visual) input, using auditory cues (volume and time difference between the two ears) to localize a target in space, while cells in the superior colliculus respond primarily to visual (but also more weakly to auditory and tactile) input. When you hear a footfall behind you and whirl around to see what it is, that’s your colliculi at work. The inferior colliculus derives from the auditory tectum of reptiles, while the superior colliculus derives from the optic tectum in reptiles, which are their primary sensory systems. In birds and mammals, both the inferior and superior colliculi sit underneath evolutionarily newer structures, the auditory and visual cortices.67 Because these colliculi are organized topographically in terms of an output, the multiple sensory systems have to be co-aligned. Likely because it is easier to calculate an orientation from visual cues than from auditory ones, the alignment of both (even the inferior colliculus) turns out to be based on visual signals.68
Eric Knudsen and his students used this phenomenon to study how the inferior colliculus of barn owls changes when the visual inputs are changed.69 Owls are particularly good at localizing sounds in three-dimensional space (since they usually hunt at night). An owl can track the footfalls of a mouse from meters away, target in on it, and hit it accurately.70 As the owl is developing, it needs to learn to align the spatial information it derives from auditory cues with the spatial information it derives from visual cues. Knudsen and colleagues fitted owls with prism glasses that shifted the visual signals to the left or the right by a certain angle.I In the owl, the auditory system shifts to align to the visual, so the auditory inputs to the inferior colliculus had to arrive at slightly different locations.
What Knudsen and colleagues found was that, initially, the auditory inputs to the inferior colliculus were very broad, but that they were pruned away as animals progressed through development. In particular, the owls passed through a specific age, called a sensitive period, after which the auditory inputs crystallized into place and became no longer malleable.72 Manipulations of the prisms before the crystallization were much more effective than manipulations afterwards.
This sort of sensitive period is seen in many systems, including the visual system, the auditory system, and others. Similar sensitive periods are seen in cats and monkeys learning to see, birds learning to sing, and human children learning language.73 This is why children are much better at learning foreign languages than adults. The timing of the sensitive periods is due to a physiological change in the brain, ending when plasticity crystallizes in different brain structures at different ages.74
But, again, just as was seen in the phantom limb results, above, the shifts that the owls could accommodate behaviorally completely matched the extent of the physical changes.75 The shifts available depended on the connectivity changes. Similar dependences have been seen in somatosensory systems and in auditory systems.76 Sensory changes reflect actual changes in physical connectivity within the brain. But what about memories? Can’t information be translated from short-term to long-term memory?
On recovery from his surgery, H.M. showed a new and profound deficit in his ability to learn new memories. However, H.M.’s earlier memories seemed to be intact. Older memories were better represented than more recent memories.77 At the time, this was taken to imply that memories were transferred from a short-term memory store to a long-term memory store (like information being written from your computer memory to a hard disk or to a CD). But further studies have shown that these older memories are different from the more recent memories.78 Something is changed in the transfer.
Long-term memories tend to be very semantic, stored as facts, as narratives, as scripts and stories, while short-term memories tend to be episodic, with a personal “I was there” emotion to them.79 This does not mean that we don’t have long-term memories with strong “I was there” emotions, but in H.M. and other similar amnesic patients, those older episodic memories vanished along with the short-term memories.80 What seems to happen to memories is that, in the short term, they are stored as episodic events, but with time, they become relegated to a semantic storage that is fundamentally different. All of H.M.’s long-term memories, for example, were semantic descriptions and did not contain the episodic descriptions that we normally associate with important long-term memories.
How might this difference between semantic and episodic memory arise? Throughout this book, I’ve tried to explain mechanism with theories that can actually explain how the information level arises from the physical. In this case, we require the answers to four questions: (1) How are episodic memories stored? (2) How are semantic memories stored? (3) How is information transferred from one to the other? In addition, it would be nice to understand (4) Why is such a mechanism evolutionarily useful?
We have encountered episodic memories elsewhere in this book (in Chapter 9). We noted that they were not stored flawlessly, but rebuilt from components stored in different cortical areas.81 The structure tying those components together seems to be the hippocampus, which receives input (through the entorhinal cortex) from pretty much the entire neocortex, and which sends output (again through the entorhinal cortex) to pretty much the entire neocortex.82 We have also encountered semantic memories elsewhere in this book (in Chapter 12).
Why would we need these two different systems? Presumably, as with the other examples of multiple decision-making systems that we’ve encountered so far, each mechanism has advantages and disadvantages. By using the right one at the right time, we can access the advantages of each and reduce the disadvantages. One theory, proposed by Jay McClelland, Randy O’Reilly, and Bruce McNaughton in the 1990s, was that we need two learning systems—one system that could store a few memories quickly by making sure the representations were really separable, and another system that could store lots of memories, but took the time to store them slowly so that they wouldn’t interfere with each other.83 The theory arose from work in the computational aspects of content-addressable memories (see Appendix C), in which it was observed that storing memories sequentially in a neural system where memories were distributed across many neurons produced catastrophic interference—as one memory was stored, older memories were lost.84
Because these systems stored information in a distributed manner, through small changes in individual connections, each connection between neurons (each synapse) participated in many memories. This meant that if you stored a memory, the changes in synapses needed to store the new memory could undo some of the changes needed for the previous memories. This is a process called interference—new memories interfere with the old ones.85 On the other hand, if you interleaved the storage of the memories, changing the synapses a little bit toward what they would need to be to store the first memory, and then a little bit toward what they would need to be to store the second, the synapses could find a way to store both memories simultaneously.
Effectively, there is a tradeoff between being malleable enough to learn new things quickly and stable enough to hold memories for a long time. McClelland, O’Reilly, and McNaughton proposed that if you had a system that could store memories quickly, without interference, then you could use that fast-storing system to interleave the memories into the slower, longer-term storage. They proposed that the hippocampus, which contains mechanisms to reduce interference between stored memories (and thus can store memories quickly), served as the fast storage and the cortex (which learned more slowly and contained many more synapses) served as the slow storage.86 Although this theory does seem to be primarily correct, as we’ve discussed in this chapter, the transfer is a transfer, not just of information but also of kind, from an episodic representation to a semantic representation.87
This means that the brain needs to transfer a memory from a quickly learned system which reconstructs it from parts into a slowly learned storage system in which it is stored in the strength of connections between neural structures.88 Before we look at how this transfer occurs, it is important to note that the evidence is very strong that these are two separate systems, each of which can learn memories separately. Although learning generally proceeds from quickly learned episodic memories to more slowly learned semantic memories, if structures critical for the episodic memory systems are damaged, the semantic memory system can still learn.89 For example, patients with hippocampal damage are impaired at episodic future thinking but are still able to learn to recognize familiar scenes that they experience multiple times and are still able to construct narratives.90 Just such an example can be found in Oliver Sacks’ lost mariner, who found a way to become comfortable with his new garden but remained unable to ever learn explicit (episodic) memories.91 Suzanne Corkin says that through her long interaction with him, H.M. began to recognize her and thought he knew her from high school.92 This construction of an explanation is sometimes called confabulation93 and is a consequence of the construction of narratives (Chapter 12) through content-addressable-memory processes (Appendix C).
Although it is possible for semantic memories to be constructed through extended experience, it is also possible to take a single experience (encoded in episodic memory) and, through internal repetition, transfer it to a semantic memory. This transfer seems to occur primarily during sleep.94 Scientists have known for a long time that sleep is a critical part of the learning process. Not only do animals and humans deprived of sleep not learn well, but they also do not easily remember the tasks that were learned before they were deprived of sleep.95
One of the most interesting phenomena discovered over the past couple of decades is a phenomenon called “replay,” in which neural firing patterns seen during behavior replay themselves during sleep afterwards.96 The phenomenon was originally seen in neural recordings of the hippocampus but is now known to occur throughout the hippocampus, neocortex, and some subcortical structures. Because it is so hard to record neural signals from humans, the replay phenomenon has generally been studied in rats and monkeys, but the timing of the replay events corresponds nicely to when sleep-deprivation studies have found that sleep is critical to consolidating memories.97
Reactivation of behavioral neural patterns during sleep was first seen in 1989 by Constantine Pavlides and Jonathan Winson, who recorded from pairs of hippocampal cells from rats.98 Hippocampal cells in rats (place cells) have the very convenient tuning function that they respond only when the rat is in a specific position in an environment (the place field of the cell). Each cell has a different place field.99 Pavlides and Winson chose place cells with nonoverlapping place fields; thus, they were able to expose the rat to the place field of one cell while not exposing it to the place field of the other. They then found that during sleep after the behavior, the cell with the place field the animal had been exposed to fired much more than the cell with the other place field—even though the animal was not in either place field when allowed to sleep.
Our ability to observe reactivation and replay took a giant step forward with the development in 1993 of multi-tetrode recording technology by Matthew Wilson and Bruce McNaughton, who brought together several technologies and were able to record from almost 150 cells simultaneously from their behaving rats.100 Because each place cell has a different place field, from the activity across the set of cells, Wilson and McNaughton were able to decode the position of the rat during behavior from its neural ensemble. (See Appendix B for a description of how this decoding process works.) From a large neural ensemble, one can also determine whether patterns observed during behavior repeat (reactivate) afterwards during sleep.101 From a large neural ensemble, one can even determine whether sequences observed during behavior repeat (replay) afterwards during sleep. A host of studies over the subsequent years, starting from Wilson and McNaughton’s follow-up paper in 1994 and culminating in dozens of papers in the past decade, have shown that what is reactivated is actually the full neural pattern—cells that were coactive during behavior remain coactive during reactivation, while cells that were not coactive are not, and that what is replayed is the actual sequences observed by the rat: cells reactivate in the same order during sleep as during behavior.102 Control studies have shown that this reactivation is a consequence of the behavior—it does not occur during sleep before the behavior, only afterwards. Reactivation and replay are also seen in neocortical systems during sleep as well. As with the hippocampal reactivation and replay, the same neural patterns observed during behavior appear afterwards during sleep. Reactivation and replay have been seen in a host of cortical systems, including the prefrontal, parietal, and even primary visual cortex.103;J
At this point the computational and neurophysiological function of replay is unknown.106 Is it to aid storage of information within the structure itself (hippocampus or cortex)?107 Is it to transfer information from the hippocampally based episodic system to the cortically based semantic system?108 Is it to erase old memories from the hippocampus while enhancing new ones?109 While these questions are still being addressed by researchers today, we have intriguing hints that all of these functions may be important. What is known is that blocking replay, either pharmacologically (through chemical means) or electrically (by stimulating the hippocampus whenever a replay event is about to happenK), disrupts memory retention.111 It is also known that replay events in hippocampus tend to correspond to reactivation events in cortical structures, and that after replay events, neural patterns in cortical structures become more tightly coupled, even between cortical structures, exactly as would be predicted by the transfer and integration hypothesis.112
Throughout this discussion, I have avoided calling these replay events “dreams,” because we don’t know if these replay events seen in animals correspond to what we experience as dreams. (We can’t ask the animal if it is dreaming, and it is very difficult to record from humans during dreams.) But, of course, it is very likely that this reactivation/replay phenomenon being studied in animals is the physical instantiation of the phenomenon we refer to as dreams. Dreams are often jumbled sequences of past experiences.113 Although the animal experiments report actual replay of direct events, in part this may be because those sequences are the easiest to recognize. Some experiments have found interleaved representations of old and new experiences in rats during REM sleep.114 During awake, resting behavior, recent experiments have found “replays” of sequences that the animal has never actually experienced, such as the sequence experienced by the animal but backwards, or chains of experienced sequences that share commonalities but have not actually been experienced together.115
In a very intriguing experiment, Robert Stickgold and colleagues trained a population of patients with hippocampal amnesia (like H.M.) to play the videogame Tetris.116 These patients can learn tasks using their nonhippocampal memory but do not remember that they have played the game before. When Stickgold and his colleagues woke these people up from sleep and asked them about their dreams, they reported seeing strange shapes falling from the sky but were terrified because they had no idea where these shapes were coming from. It seems likely that the cortex was replaying the sensory stimuli it had seen (falling shapes) but that, without a hippocampus, these patients could not integrate these events into their episodic memories or remember where these images came from.
So even after lesions, animals (including humans) can learn new tasks, but these new tasks are learned in different ways, using different decision-making systems. Similarly, intact cortical structures can take over for damaged cortical structures, but only to the extent that input connections are available. And memories can be transferred from hippocampal-dependent episodic structures to cortical-dependent semantic processes, but they are modified in that transition. The brain is a physical machine. The mind is not software that happens to be implemented on the brain’s hardware, but is directly dependent on the processes of the physical brain.
The mind used to be thought of as a hiker through a forest, or a surfer on a wave, or perhaps as the wave itself flowing through a physical ocean, or as something affected by but separate from the brain itself.117 Historically, this separation was based on Descartes’ dualism between the physical and cognitive entities.118 In the modern cognitive science of the past century, this separation was based on the new theories of digital information.119
However, as we have seen in this chapter, the theory that mind and brain are separable is untenable, and the available data suggest instead that they are the same thing. The evidence for mentation as the processing of information is overwhelming, but the different components of the brain process that information differently. It is not a wave flowing through an open ocean, capable of traveling in any direction, but rather a wave traveling through a series of highly restrictive canyons. This has implications for every aspect of our mental lives, from mental processes like imagination to diseases and mental disorders like Parkinson’s disease, Alzheimer’s disease, post-traumatic stress disorder, and addiction, to mental constructs of behavior like craving, impulsivity, free will, and morality.
• V. S. Ramachandran and Sandra Blakeslee (1999). Phantoms in the Brain: Probing the Mysteries of the Human Mind. New York: Harper Perennial.
• Patricia S. Churchland and Terrence J. Sejnowski (1994). The Computational Brain. Cambridge, MA: MIT Press.
• Suzanne Corkin (2002). What’s new with the amnesic patient H.M.? Nature Reviews Neuroscience, 3, 153–160.
• Douglas R. Hofstadter (1979). Gödel, Escher, Bach: An Eternal Golden Braid. New York: Basic Books.
• Douglas R. Hofstadter (1985). Metamagical Themas: Questing for the Essence of Mind and Pattern. New York: Basic Books.