What the brain does may seem obvious to you. The brain gets inputs from its sensors, it processes those inputs, and then it acts. In the end, how an animal reacts to what it senses determines its success or failure. A direct mapping from sensory input to action certainly applies to some parts of the brain. For example, accidentally touching a hot surface will cause a reflex retraction of the arm. The input-output circuit responsible is located in the spinal cord. But what about the neocortex? Can we say that the task of the neocortex is to take inputs from sensors and then immediately act? In short, no.
You are reading or listening to this book and it isn’t causing any immediate actions other than perhaps turning pages or touching a screen. Thousands of words are streaming into your neocortex and, for the most part, you are not acting on them. Maybe later you will act differently for having read this book. Perhaps you will have future conversations about brain theory and the future of humanity that you would not have had if you didn’t read this book. Perhaps your future thoughts and word choices will be subtly influenced by my words. Perhaps you will work on creating intelligent machines based on brain principles, and my words will inspire you in this direction. But right now, you are just reading. If we insist on describing the neocortex as an input-output system, then the best we could say it is that the neocortex gets lots of inputs, it learns from these inputs, and then, later—maybe hours, maybe years—it acts differently based on these prior inputs.
From the moment I became interested in how the brain worked, I realized that thinking of the neocortex as an input-leads-to-output system would not be fruitful. Fortunately, when I was a graduate student at Berkeley I had an insight that led me down a different and more successful path. I was at home, working at my desk. There were dozens of objects on the desk and in the room. I realized that if any one of these objects changed, in even the slightest way, I would notice it. My pencil cup was always on the right side of the table; if one day I found it on the left, I would notice the change and wonder how it got moved. If the stapler changed in length, I would notice. I would notice the change if I touched the stapler or if I looked at it. I would even notice if the stapler made a different sound when being used. If the clock on the wall changed its location or style, I would notice. If the cursor on my computer screen moved left when I moved the mouse to the right, I would immediately realize something was wrong. What struck me was that I would notice these changes even if I wasn’t attending to these objects. As I looked around the room, I didn’t ask, “Is my stapler the correct length?” I didn’t think, “Check to make sure the clock’s hour hand is still shorter than the minute hand.” Changes to the normal would just pop into my head, and my attention would then be drawn to them. There were literally thousands of possible changes in my environment that my brain would notice almost instantly.
There was only one explanation I could think of. My brain, specifically my neocortex, was making multiple simultaneous predictions of what it was about to see, hear, and feel. Every time I moved my eyes, my neocortex made predictions of what it was about to see. Every time I picked something up, my neocortex made predictions of what each finger should feel. And every action I took led to predictions of what I should hear. My brain predicted the smallest stimuli, such as the texture of the handle on my coffee cup, and large conceptual ideas, such as the correct month that should be displayed on a calendar. These predictions occurred in every sensory modality, for low-level sensory features and high-level concepts, which told me that every part of the neocortex, and therefore every cortical column, was making predictions. Prediction was a ubiquitous function of the neocortex.
At that time, few neuroscientists described the brain as a prediction machine. Focusing on how the neocortex made many parallel predictions would be a novel way to study how it worked. I knew that prediction wasn’t the only thing the neocortex did, but prediction represented a systemic way of attacking the cortical column’s mysteries. I could ask specific questions about how neurons make predictions under different conditions. The answers to these questions might reveal what cortical columns do, and how they do it.
To make predictions, the brain has to learn what is normal—that is, what should be expected based on past experience. My previous book, On Intelligence, explored this idea of learning and prediction. In the book, I used the phrase “the memory prediction framework” to describe the overall idea, and I wrote about the implications of thinking about the brain this way. I argued that by studying how the neocortex makes predictions, we would be able to unravel how the neocortex works.
Today I no longer use the phrase “the memory prediction framework.” Instead, I describe the same idea by saying that the neocortex learns a model of the world, and it makes predictions based on its model. I prefer the word “model” because it more precisely describes the kind of information that the neocortex learns. For example, my brain has a model of my stapler. The model of the stapler includes what the stapler looks like, what it feels like, and the sounds it makes when being used. The brain’s model of the world includes where objects are and how they change when we interact with them. For example, my model of the stapler includes how the top of the stapler moves relative to the bottom and how a staple comes out when the top is pressed down. These actions may seem simple, but you were not born with this knowledge. You learned it at some point in your life and now it is stored in your neocortex.
The brain creates a predictive model. This just means that the brain continuously predicts what its inputs will be. Prediction isn’t something that the brain does every now and then; it is an intrinsic property that never stops, and it serves an essential role in learning. When the brain’s predictions are verified, that means the brain’s model of the world is accurate. A mis-prediction causes you to attend to the error and update the model.
We are not aware of the vast majority of these predictions unless the input to the brain does not match. As I casually reach out to grab my coffee cup, I am not aware that my brain is predicting what each finger will feel, how heavy the cup should be, the temperature of the cup, and the sound the cup will make when I place it back on my desk. But if the cup was suddenly heavier, or cold, or squeaked, I would notice the change. We can be certain that these predictions are occurring because even a small change in any of these inputs will be noticed. But when a prediction is correct, as most will be, we won’t be aware that it ever occurred.
When you are born, your neocortex knows almost nothing. It doesn’t know any words, what buildings are like, how to use a computer, or what a door is and how it moves on hinges. It has to learn countless things. The overall structure of the neocortex is not random. Its size, the number of regions it has, and how they are connected together is largely determined by our genes. For example, genes determine what parts of the neocortex are connected to the eyes, what other parts are connected to the ears, and how those parts connect to each other. Therefore, we can say that the neocortex is structured at birth to see, hear, and even learn language. But it is also true that the neocortex doesn’t know what it will see, what it will hear, and what specific languages it might learn. We can think of the neocortex as starting life having some built-in assumptions about the world but knowing nothing in particular. Through experience, it learns a rich and complicated model of the world.
The number of things the neocortex learns is huge. I am sitting in a room with hundreds of objects. I will randomly pick one: a printer. I have learned a model of the printer that includes it having a paper tray, and how the tray moves in and out of the printer. I know how to change the size of the paper and how to unwrap a new ream and place it in the tray. I know the steps I need to take to clear a paper jam. I know that the power cord has a D-shaped plug at one end and that it can only be inserted in one orientation. I know the sound of the printer and how that sound is different when it is printing on two sides of a sheet of paper rather than on one. Another object in my room is a small, two-drawer file cabinet. I can recall dozens of things I know about the cabinet, including what is in each drawer and how the objects in the drawer are arranged. I know there is a lock, where the key is, and how to insert and turn the key to lock the cabinet. I know how the key and lock feel and the sounds they make as I use them. The key has a small ring attached to it and I know how to use my fingernail to pry open the ring to add or remove keys.
Imagine going room to room in your home. In each room you can think of hundreds of things, and for each item you can follow a cascade of learned knowledge. You can also do the same exercise for the town you live in, recalling what buildings, parks, bike racks, and individual trees exist at different locations. For each item, you can recall experiences associated with it and how you interact with it. The number of things you know is enormous, and the associated links of knowledge seem never-ending.
We learn many high-level concepts too. It is estimated that each of us knows about forty thousand words. We have the ability to learn spoken language, written language, sign language, the language of mathematics, and the language of music. We learn how electronic forms work, what thermostats do, and even what empathy or democracy mean, although our understanding of these may differ. Independent of what other things the neocortex might do, we can say for certain that it learns an incredibly complex model of the world. This model is the basis of our predictions, perceptions, and actions.
The inputs to the brain are constantly changing. There are two reasons why. First, the world can change. For example, when listening to music, the inputs from the ears change rapidly, reflecting the movement of the music. Similarly, a tree swaying in the breeze will lead to visual and perhaps auditory changes. In these two examples, the inputs to the brain are changing from moment to moment, not because you are moving but because things in the world are moving and changing on their own.
The second reason is because we move. Every time we take a step, move a limb, move our eyes, tilt our head, or utter a sound, the input from our sensors change. For example, our eyes make rapid movements, called saccades, about three times a second. With each saccade, our eyes fixate on a new point in the world and the information from the eyes to the brain changes completely. This change would not occur if we hadn’t moved our eyes.
The brain learns its model of the world by observing how its inputs change over time. There isn’t another way to learn. Unlike with a computer, we cannot upload a file into our brain. The only way for a brain to learn anything is via changes in its inputs. If the inputs to the brain were static, nothing could be learned.
Some things, like a melody, can be learned without moving the body. We can sit perfectly still, with eyes closed, and learn a new melody by just listening to how the sounds change over time. But most learning requires that we actively move and explore. Imagine you enter a new house, one you have not been in before. If you don’t move, there will be no changes in your sensory input, and you can’t possibly learn anything about the house. To learn a model of the house, you have to look in different directions and walk from room to room. You need to open doors, peek in drawers, and pick up objects. The house and its contents are mostly static; they don’t move on their own. To learn a model of a house, you have to move.
Take a simple object such as a computer mouse. To learn what a mouse feels like, you have to run your fingers over it. To learn what a mouse looks like, you have to look at it from different angles and fixate your eyes on different locations. To learn what a mouse does, you have to press down on its buttons, slide off the battery cover, or move it across a mouse pad to see, feel, and hear what happens.
The term for this is sensory-motor learning. In other words, the brain learns a model of the world by observing how our sensory inputs change as we move. We can learn a song without moving because, unlike the order in which we can move from room to room in a house, the order of notes in a song is fixed. But most of the world isn’t like that; most of the time we have to move to discover the structure of objects, places, and actions. With sensory-motor learning, unlike a melody, the order of sensations is not fixed. What I see when I enter a room depends on which direction I turn my head. What my finger feels when holding a coffee cup depends on whether I move my finger up or down or sideways.
With each movement, the neocortex predicts what the next sensation will be. Move my finger up on the coffee cup and I expect to feel the lip, move my finger sideways and I expect to feel the handle. If I turn my head left when entering my kitchen, I expect to see my refrigerator, and if I turn my head right, I expect to see the range. If I move my eyes to the left front burner, I expect to see the broken igniter that I need to fix. If any input doesn’t match the brain’s prediction—perhaps my spouse fixed the igniter—then my attention is drawn to the area of mis-prediction. This alerts the neocortex that its model of that part of the world needs to be updated.
The question of how the neocortex works can now be phrased more precisely: How does the neocortex, which is composed of thousands of nearly identical cortical columns, learn a predictive model of the world through movement?
This is the question my team and I set out to answer. Our belief was that if we could answer it, we could reverse engineer the neocortex. We would understand both what the neocortex did and how it did it. And ultimately, we would be able to build machines that worked the same way.
Before we can start answering the question above, there are a few more basic ideas you need to know. First, like every other part of the body, the brain is composed of cells. The brain’s cells, called neurons, are in many ways similar to all our other cells. For example, a neuron has a cell membrane that defines its boundary and a nucleus that contains DNA. However, neurons have several unique properties that don’t exist in other cells in your body.
The first is that neurons look like trees. They have branch-like extensions of the cell membrane, called axons and dendrites. The dendrite branches are clustered near the cell and collect the inputs. The axon is the output. It makes many connections to nearby neurons but often travels long distances, such as from one side of the brain to the other or from the neocortex all the way down to the spinal cord.
The second difference is that neurons create spikes, also called action potentials. An action potential is an electrical signal that starts near the cell body and travels along the axon until it reaches the end of every branch.
The third unique property is that the axon of one neuron makes connections to the dendrites of other neurons. The connection points are called synapses. When a spike traveling along an axon reaches a synapse, it releases a chemical that enters the dendrite of the receiving neuron. Depending on which chemical is released, it makes the receiving neuron more or less likely to generate its own spike.
Considering how neurons work, we can state two fundamental tenets. These tenets will play important roles in our understanding of the brain and intelligence.
At any point in time, some neurons in the neocortex are actively spiking and some are not. Typically, the number of neurons that are active at the same time is small, maybe 2 percent. Your thoughts and perceptions are determined by which neurons are spiking. For example, when doctors perform brain surgery, they sometimes need to activate neurons in an awake patient’s brain. They stick a tiny probe into the neocortex and use electricity to activate a few neurons. When they do this, the patient might hear, see, or think something. When the doctor stops the stimulation, whatever the patient was experiencing stops. If the doctor activates different neurons, the patient has a different thought or perception.
Thoughts and experiences are always the result of a set of neurons that are active at the same time. Individual neurons can participate in many different thoughts or experiences. Every thought you have is the activity of neurons. Everything you see, hear, or feel is also the activity of neurons. Our mental states and the activity of neurons are one and the same.
The brain remembers a lot of things. You have permanent memories, such as where you grew up. You have temporary memories, such as what you had for dinner last night. And you have basic knowledge, such as how to open a door or how to spell the word “dictionary.” All these things are stored using synapses, the connections between neurons.
Here is the basic idea for how the brain learns: Each neuron has thousands of synapses, which connect the neuron to thousands of other neurons. If two neurons spike at the same time, they will strengthen the connection between them. When we learn something, the connections are strengthened, and when we forget something, the connections are weakened. This basic idea was proposed by Donald Hebb in the 1940s and today it is referred to as Hebbian learning.
For many years, it was believed that the connections between neurons in an adult brain were fixed. Learning, it was believed, involved increasing or decreasing the strength of synapses. This is still how learning occurs in most artificial neural networks.
However, over the past few decades, scientists have discovered that in many parts of the brain, including the neocortex, new synapses form and old ones disappear. Every day, many of the synapses on an individual neuron will disappear and new ones will replace them. Thus, much of learning occurs by forming new connections between neurons that were previously not connected. Forgetting happens when old or unused connections are removed entirely.
The connections in our brain store the model of the world that we have learned through our experiences. Every day we experience new things and add new pieces of knowledge to the model by forming new synapses. The neurons that are active at any point in time represent our current thoughts and perceptions.
We have now gone over several of the basic building blocks of the neocortex—some of the pieces of our puzzle. In the next chapter, we start putting these pieces together to reveal how the entire neocortex works.