9 Understanding the Brain

For many reasons, understanding the mechanisms that underlie the development and the plasticity phenomena of the human brain would be of enormous value. From a medical standpoint, the understanding of these mechanisms would lead to ways to prevent degenerative brain disorders, to extend life, and to increase the length of time people live in full possession of their mental abilities. By themselves, those objectives would justify the enormous effort now under way to understand in detail how the brain operates. However, such an understanding would also be valuable from an engineering standpoint, because deep knowledge of the mechanisms by which the brain operates would also give us the tools necessary to recreate the same behaviors in a computer.

To advance our understanding of the brain, we need to obtain extensive and detailed information about its internal organization and about how its activation patterns encode thoughts. Advances in instrumentation technology have given us many ways to observe a living brain. The objective is to improve our knowledge of the structures and mechanisms involved in the creation and operation of a brain without disturbing its behavior. Techniques that involve the destruction of the brain can provide much finer complementary information about neuron structure and connectivity.

A number of very large research projects and a multitude of smaller ones aim at obtaining information about brain structure and behavior that can be used to reveal how brains work, but also to understand a number of other important things.

In the United States, the multi-million-dollar Human Connectome Project (HCP) aims to provide an extensive compilation of neural data and a graphic user interface for navigating through the data. The objective is to obtain new knowledge about the living human brain by building a “network map” that will provide information on the anatomical and functional connectivity within a human brain and to use this knowledge to advance our understanding of neurological disorders. Another major project, the Open Connectome Project, aims at creating publicly available connectome data from many organisms.

Also in the United States, the Brain Research through Advancing Innovative Neurotechnologies (BRAIN) initiative, also known as the Brain Activity Map Project, is a multi-billion-dollar research initiative announced in 2013, with the goal of mapping the activity of every neuron in the human brain. The idea is that, by accelerating the development and application of new technologies, it will be possible to obtain a new dynamic view of the brain that will show, for the first time, how individual cells and complex neural circuits interact.

In Europe, the Human Brain Project (HBP) is a ten-year project, with a budget of more than a billion euros, financed largely by the European Union. Established in 2013, it is coordinated by the École Polytechnique Fédérale de Lausanne, the same university that coordinated the Blue Brain Project, a project that obtained some of the most significant research results to date in brain simulation. The Human Brain Project aims to achieve a unified, multi-level, understanding of the human brain by integrating data about healthy and diseased brains. The project focuses on the data that will have to be acquired, stored, organized, and mined in order to identify relevant features in the brain. One of its main objectives is the development of novel neuromorphic and neuro-robotic technologies based on the brain’s circuitry and computing principles.

In Japan, Brain/MINDS (Brain Mapping by Integrated Neurotechnologies for Disease Studies) is a multi-million-dollar project, launched in 2014, that focuses on using non-human primate brains to obtain a better understanding of the workings of the human brain. As is also true of the other projects, one of the principal aims of Brain/MINDS is to elucidate the mechanisms involved in brain disorders. The idea is to use the marmoset, a small primate, as a model to study cognition and neural mechanisms that lead to brain disorders. (Simpler model organisms, such as the mouse, may be too far away evolutionarily from humans to provide adequate platforms to study the human brain. Using marmosets may help circumvent this limitation.)

The Allen Institute for Brain Science is a nonprofit private research organization that conducts a number of projects aimed at understanding how the human brain works. The Allen Human Brain Atlas (Hawrylycz et al. 2012), now under development, is a highly comprehensive information system that integrates data collected by means of live brain imaging, tissue microscopy, and DNA sequencing to document many different pieces of information about the brain of mice, non-humans primates, and humans. The information made available includes where in the brain certain genes are active, brain connectivity data, and data about the morphology and behavior of specific neuron cells.

Looking Inside

The above-mentioned projects, and many other projects that address similar matters, use various technologies to derive detailed information about the brains of humans, primates, and other model organisms. Common to many of these projects are the techniques used to obtain information about brain structures and even, in some cases, about neuron-level connectivity.

In this section we will consider imaging methods that can be used, in non-invasive ways, to obtain information about working brains in order to observe their behavior. Such methods are known as neuroimaging. When the objective is to obtain detailed information about the three-dimensional structure and composition of brain tissues, it is useful to view the brain as divided into a large number of voxels. A voxel is the three-dimensional equivalent of a pixel in a two-dimensional image. More precisely, a voxel is a three-dimensional rectangular cuboid that corresponds to the fundamental element of a volume, with the cuboid dimensions imposed by the imaging technology. Cuboid edge sizes range from just a few nanometers to a centimeter or more, depending on the technology and the application. Smaller voxels contain fewer neurons on average, and correspond to lower levels of neuronal activity. Therefore, the smaller the voxels, the harder it is to obtain accurate information about their characteristics using imaging techniques that look at the levels of electrical or chemical activity. Smaller voxels also take longer to scan, since scanning time, in many technologies, increases with the number of voxels. With existing technologies, a voxel used in the imaging of live brains will typically contain a few million neurons and a few billion synapses, the actual number depending on the voxel size and the region of the brain being imaged. Voxels are usually arranged in planes, or slices, which are juxtaposed to obtain complete three-dimensional information about the brain. Many of the techniques used to image the brain are also used to image other parts of the body, although some are specifically tuned to the particular characteristics of brain tissue.

Neuroimaging uses many different physical principles to obtain detailed information about the structure and behavior of the brain. Neuroimaging can be broken into two large classes: structural imaging (which obtains information about the structure of the brain, including information about diseases that manifest themselves by altering the structure) and functional imaging (which obtains information about brain function, including information that can be used to diagnose diseases affecting function, perhaps without affecting large structures).

Among the technologies that have been used in neuroimaging (Crosson et al. 2010) are computed tomography (CT), near-infrared spectroscopy (NIRS), positron-emission tomography (PET), a number of variants of magnetic-resonance imaging (MRI), electroencephalography (EEG), magnetoencephalography (MEG), and event-related optical signal (EROS).

Computed tomography (Hounsfield 1973), first used by Godfrey Hounsfield in 1971 at Atkinson Morley’s Hospital in London, is an imaging technique that uses computer-processed x-ray images to produce tomographic images which are virtual slices of tissues that, stacked on top of each other, compose a three-dimensional image. X rays with frequencies between 30 petahertz (3 × 1016 Hz) and 30 exahertz (3 × 1019 Hz) are widely used to image the insides of objects, since they penetrate deeply in animal tissues but are attenuated to different degrees by different materials.

Tomographic images enable researchers to see inside the brain (and other tissues) without cutting. Computer algorithms process the received x-ray images and generate a three-dimensional image of the inside of the organ from a series of two-dimensional images obtained by sensors placed outside the body. Usually the images are taken around a single axis of rotation; hence the term computed axial tomography (CAT). Computed tomography data can be manipulated by a computer in order to highlight the different degrees of attenuation of an x-ray beam caused by various body tissues. CT scans may be done with or without the use of a contrast agent (a substance that, when injected into the organism, causes a particular organ or tissue to be seen more clearly with x rays). The use of contrast dye in CT angiography gives good visualization of the vascular structures in the blood vessels in the brain.

Whereas x-ray radiographs have resolutions comparable to those of standard photographs, computed tomography only reaches a spatial resolution on the order of a fraction of a millimeter (Hsieh 2009). A CT scan takes only a few seconds but exposes the subject to potentially damaging ionizing radiation. Therefore, CT scans are not commonly used to study brain behavior, although they have provided important information about brain macro-structures. CT scans are used mostly to determine changes in brain structures that occur independently of the level of activity.

Because the characteristics of the tissue change slightly when the neurons are firing, it is possible to visualize brain activity using imaging techniques. One of the most significant effects is the change in blood flow. When a specific area of the brain is activated, the blood volume in the area changes quickly. A number of imaging techniques, including NIRS, PET, and MRI, use changes in blood volume to detect brain activity. The fact that water, oxygenated hemoglobin, and deoxygenated hemoglobin absorb visible and near-infrared light can also be used to obtain information about the location of neuronal activity.

Blood oxygenation varies with levels of neural activity because taking neurons back to their original polarized state after they have become active and fired requires pumping ions back and forth across the neuronal cell membranes, which consumes chemical energy. The energy required to activate the ion pumps is produced mainly from glucose carried by the blood. More blood flow is necessary to transport more glucose, also bringing in more oxygen in the form of oxygenated hemoglobin molecules in red blood cells. The blood-flow increase happens within 2–3 millimeters of the active neurons. Usually the amount of oxygen brought in exceeds the amount of oxygen consumed in burning glucose, which causes a net decrease in deoxygenated hemoglobin in that area of a brain. Although one might expect blood oxygenation to decrease with activation, the dynamics are a bit more complex than that. There is indeed a momentary decrease in blood oxygenation immediately after neural activity increases, but it is followed by a period during which the blood flow increases, overcompensating for the increased demand, and blood oxygenation actually increases after neuronal activation. That phenomenon, first reported by Seiji Ogawa, is known as the blood-oxygenation-level-dependent (BOLD) effect. It changes the properties of the blood near the firing neurons. Because it provides information about the level of blood flow in different regions of the brain, it can be used to detect and monitor brain activity (Ogawa et al. 1990). The magnitude of the BOLD signal peaks after a few seconds and then falls back to a base level. There is evidence that the BOLD signal is more closely related to the input than to the output activity of the neurons in the region (Raichle and Mintun 2006). In parts of the cortex where the axons are short and their ends are near the neuron bodies, it makes no difference whether the BOLD signal is correlated with the input or with the output of the neurons, since the voxels are not small enough to distinguish between the different parts of the neuron. In other areas of the brain, where the axons are longer, the difference between input and output activity can be significant.

Obtaining accurate measures of the level of the BOLD signal is difficult, since the signal is weak and can be corrupted by noise from many sources. In practice, sophisticated statistical procedures are required to recover the underlying signal. The resulting information about brain activation can be viewed graphically by color coding the levels of activity in the whole brain or in the specific region being studied. By monitoring the BOLD signal, it is possible to localize activity to within millimeters with a time resolution of a few seconds. Alternative technologies that can improve both spatial resolution and time resolution through the use of biomarkers other than the BOLD signal are under development, but they have other limitations. Therefore, the majority of techniques used today use the BOLD signal as a proxy for brain activity.

Near-infrared spectroscopy (NIRS) is a technique based on the use of standard electromagnetic radiation that uses a different part of the spectrum than CT techniques use: the range from 100 to 400 terahertz (that is, from 1 × 1014 to 4 × 1014 Hz). NIRS can be used to study the brain because transmission and absorption of NIR photons by body tissues reveal information about changes in hemoglobin concentration (Villringer et al. 1993). NIRS can be used non-invasively to monitor brain function by measuring the BOLD signal because in the NIRS frequency range light may diffuse several centimeters through the tissue before it is diffused and detected (Boas, Dale, and Franceschini 2004). A NIRS measurement consists in sending photons of appropriate frequency into the human brain, sensing the diffused light, and using computer algorithms to compute the densities of substances causing the photons to diffuse.

NIRS is sensitive to the volume of tissue residing between the source of light entering the tissue and the detector receiving the light that diffuses out of the tissue. Since NIR light penetrates only a few centimeters into the human brain before being diffused, the source and the detector are typically placed on the scalp, separated by a few centimeters. The resulting signal can be used to image mainly the most superficial cortex. NIRS is a non-invasive technique that can be used to measure hemodynamic signals with a temporal resolution of 100 Hz or better, although it is always limited by the slow response of the BOLD effect. Functional NIR imaging (fNIR) has several advantages in cost and portability over MRI and other techniques, but it can’t be used to measure cortical activity more than a few centimeters deep in the skull, and it has poorer spatial resolution. The use of NIRS in functional mapping of the human cortex is also called diffuse optical tomography (DOT).

Positron-emission tomography (PET) is a computerized imaging technique that uses the particles emitted by unstable isotopes that have been injected into the blood. The technique is based on work done by David Kuhl, Luke Chapman, and Roy Edwards in the 1950s at the University of Pennsylvania. In 1953, Gordon Brownell, Charles Burnham, and their group at Massachusetts General Hospital demonstrated the first use of the technology for medical imaging (Brownell and Sweet 1953).

PET is based on the detection of pairs of high-energy photons emitted in the decay of a positron emitted by a radioactive nucleus that has been injected into the body as part of a biologically active molecule. A positron is a sub-atomic particle with the same mass as an electron, but with positive charge. It is, in fact, a piece of antimatter—an anti-electron. When emitted from a decaying radionuclide integrated in ordinary matter, a positron can travel only a very short distance before encountering an electron, an event that annihilates both particles and results in two high-energy photons (511 KeV, or kilo-electron-volts) leaving the site of annihilation and traveling in opposite directions, almost exactly 180 degrees from one another. Three-dimensional images of the concentration of the original radionuclide (the tracer) within the body are then constructed by means of automated computer analysis.

The biologically active molecule most often chosen for use in PET is fluorodeoxyglucose (FDG), an analogue of glucose. When it is used, the concentrations of tracer imaged are correlated with tissue metabolic activity, which again involves increased glucose uptake and the BOLD signal. The most common application of PET, detection of cancer tissues, works well with FDG because cancer tissues, owing to their differences in structure from non-cancerous tissues, produce visible signatures in PET images.

When FDG is used in imaging, the normal fluorine atom in each FDG molecule is replaced by an atom of the radioactive isotope fluorine-18. In the decay process, known as β+ decay, a proton is replaced by a neutron in the nucleus, and the nucleus emits a positron and an electron neutrino. (See figure 9.1.) This isotope, 18F, has a half-life (meaning that half of the fluorine-18 atoms will have emitted one positron and decayed into stable oxygen-18 atoms) of 110 minutes.

10998_009_fig_001.jpg

Figure 9.1 A schematic depiction of the decay process by which a fluorine-18 nucleus leads to the production of two high-energy photons (drawing not to scale).

One procedure used in neuroimaging involves injecting labeled FDG into the bloodstream and waiting about half an hour so that the FDG not used by neurons leaves the brain. The labeled FDG that remains in the brain tissues has been metabolically trapped within the tissue, and its concentration gives an indication of the regions of the brain that were active during that time. Therefore, the distribution of FDG in brain tissue, measured by the number of observed positron decays, can be a good indicator of the level of glucose metabolism, a proxy for neuronal activity. However, the time scales involved are not useful for studying brain activity related to changes in mental processes lasting only a few seconds.

However, the BOLD effect can be used in PET to obtain evidence of brain activity with higher temporal resolution. Other isotopes with half-lives shorter than that of FDG—among them carbon-11 (half-life 20 minutes), nitrogen-13 (half-life 10 minutes), and oxygen-15 (half-life 122 seconds) are used, integrated into a large number of different active molecules. The use of short-lived isotopes requires that a cyclotron be nearby to generate the unstable isotope so that it can be used before a significant fraction decays.

Despite the relevance of the previous techniques, the technique most commonly used today to study macro-scale brain behavior is magnetic-resonance imaging. Paul Lauterbur of the State University of New York at Stony Brook developed the theory behind MRI (Lauterbur 1973), building on previous work by Raymond Damadian and Herman Carr.

MRI is based on a physical phenomenon that happens when hydrogen nuclei are exposed to electric and magnetic fields. Hydrogen nuclei, which are in fact protons, have an intrinsic property, called nuclear spin, that makes them behave like small magnets that align themselves parallel to an applied magnetic field. When such a field is applied, a small fraction of the nuclei of atoms of hydrogen present, mostly in water molecules, align with the applied field. They converge to that alignment after going through a decaying oscillation of a certain frequency, called the resonating frequency, which depends on the intensity of the field. For a magnetic field of one tesla (T), the resonating frequency is 42 megahertz and it increases linearly with the strength of the field. Existing equipment used for human body imaging works with very strong magnetic fields (between 1.5 and 7 teslas). For comparison, the strength of the Earth’s magnetic field at sea level ranges from 25 to 65 microteslas, a value smaller by a factor of about 100,000.

If one applies a radio wave of a frequency close enough to the resonating frequency of these nuclei, the alignment deviates from the applied magnetic field, much as a compass needle would deviate from the north-south line if small pushes of the right frequency were to be applied in rapid succession. When this excitation process terminates, the nuclei realign themselves with the magnetic field, again oscillating at their specific resonating frequency. While doing this, they emit radio waves that can be detected and processed by computer and then used to create a three-dimensional image in which different types of tissues can be distinguished by their structure and their water content.

In practice, it isn’t possible to detect with precision the position of an atom oscillating at 42 megahertz, or some frequency of the same order, because the length of the radio waves emitted is too long. Because of physical limitations, we can only detect the source of a radio wave with an uncertainty on the order of its wavelength, which is, for the radio waves we are considering here, several meters. However, by modulating the magnetic field and changing its strength with time and along the different dimensions of space, the oscillating frequency of the atoms can be finely controlled to depend on their specific positions. This modulation causes each nucleus to emit at a specific frequency that varies with its location, and also with time, in effect revealing its whereabouts to the detectors. Figure 9.2 illustrates how control of the magnetic field can be used to pinpoint the locations of oscillating hydrogen nuclei.

10998_009_fig_002.jpg

Figure 9.2 An illustration of how protons, oscillating at frequencies that depend on the strength of the magnetic field, make it possible to determine the precise locations of hydrogen atoms.

MRI has some limitations in terms of its space and time resolution. Because a strong enough signal must be obtained, the voxels cannot be too small. The stronger the magnetic field, the smaller the voxels can be, but even 7-tesla MRI machines cannot obtain high-quality images with voxels much smaller than about one cubic millimeter. However, as technology evolves, one can hope that this resolution will improve, enabling MRI to obtain images with significantly smaller voxels.

The physical principles underlying MRI can be used in accordance with many different protocols to obtain different sorts of information. In its simplest and most commonly used form, MRI is used to obtain static images of the brain for the purpose of diagnosing tumors or other diseases, which manifest themselves as changes in the brain’s macro-structures. These changes become visible because different tissues have different MRI signatures. Figure 9.3 shows the images of three slices of one brain. Image a corresponds to a slice near the top of the skull, with the cortex folds clearly visible. Image b shows some additional cortex folds; also visible are the lateral ventricles, where cerebrospinal fluid is produced, and the corpus callosum, the largest white matter structure in the brain. Image c shows a clearly abnormal structure (a benign brain tumor) on the right side of the image, near the center.

10998_009_fig_003.jpg

Figure 9.3 Images of brain slices obtained with magnetic-resonance imaging.

MRI techniques can also be used to obtain additional information about brain behavior and structure. Two important techniques are functional MRI and diffusion MRI.

In recent decades, functional MRI (fMRI) has been extensively used in brain research. Functional MRI was first proposed in 1991 by Jack Belliveau, who was working at the Athinoula A. Martinos Center, in Boston. (See Belliveau et al. 1991.) Belliveau used a contrast agent injected in the bloodstream to obtain a signal that could be correlated with the levels of brain activity in particular areas. In 1992, Kenneth Kwong (Martinos Center), Seiji Ogawa (AT&T Bell Laboratories), and Peter Bandettini (Medical College of Wisconsin) reported that the BOLD signal could be used directly in fMRI as a proxy for brain activity.

The BOLD effect can be used in fMRI because the difference in the magnetic properties of oxygen-rich and oxygen-poor hemoglobin leads to differences in the magnetic-resonance signals of oxygenated and deoxygenated blood. The magnetic resonance is stronger where blood is more highly oxygenated and weaker where it is not. The time scales involved in the BOLD effect largely define the temporal resolution. For fMRI, the hemodynamic response lasts more than 10 seconds, rising rapidly, peaking at 4 to 6 seconds, and then falling exponentially fast. The signal detected by fMRI lags the neuronal events triggering it by a second or two, because it takes that long for the vascular system to respond to the neuron’s need for glucose.

The time resolution of fMRI is sufficient for studying a number of brain processes. Neuronal activities take anywhere from 100 milliseconds to a few seconds. Higher reasoning activities, such as reading or talking, may take anywhere from a few seconds to many minutes. With a time resolution of a few seconds, most fMRI experiments study brain processes lasting from a few seconds to minutes, but because of the need for repetition, required to improve the signal-to-noise ratio, the experiments may last anywhere from a significant fraction of an hour to several hours,.

Diffusion MRI (dMRI) is another method that produces magnetic-resonance images of the structure of biological tissues. In this case, the images represent the local characteristics of molecular diffusion, generally of molecules of water (Hagmann et al. 2006). Diffusion MRI was first proposed, by Denis Le Bihan, in 1985. The technique is based on the fact that MRI can be made sensitive to the motion of molecules, so that it can be used to show contrast related to the structure of the tissues at microscopic level (Le Bihan and Breton 1985). Two specific techniques used in dMRI are diffusion weighted imaging (DWI) and diffusion tensor imaging (DTI).

Diffusion weighted imaging obtains images whose intensity correlates with the random Brownian motion of water molecules within a voxel of tissue. Although the relationship between tissue anatomy and diffusion is complex, denser cellular tissues tend to exhibit lower diffusion coefficients, and thus DWI can be used to detect certain types of tumors and other tissue malformations.

Tissue organization at the cellular level also affects molecule diffusion. The structural organization of the white matter of the brain (composed mainly of glial cells and of myelinated axons, which transmit signals from one region of the brain to another) can be studied in some detail by means of diffusion MRI because diffusion of water molecules takes places preferentially in some directions. Bundles of axons make the water diffuse preferentially in a direction parallel to the direction of the fibers.

Diffusion tensor imaging uses the restricted directions of diffusion of water in neural tissue to produce images of neural tracts. In DTI, to each voxel corresponds one ellipsoid, whose dimensions give the intensity and the directions of the diffusion in the voxel. The ellipsoid can be characterized by a matrix (a tensor). The directional information at each voxel can be used to identify neural tracts. By a process called tractography, DTI data can be used to represent neural tracts graphically. By color coding individual tracts, it is possible to obtain beautiful and highly informative images of the most important connections between brain regions.

Tractography is used to obtain large-scale, low-resolution information about the connection patterns of the human brain. Figure 9.4 shows an image obtained by means of diffusion tensor imaging of the mid-sagittal plane, the plane that divides the brain into a left and a right side. The axon fibers connecting different regions of the brain, including the fibers in the corpus callosum connecting the two hemispheres and crossing the mid-sagittal plane, are clearly visible.

10998_009_fig_004.jpg

Figure 9.4 DTI reconstruction of tracts of brain fiber that run through the mid-sagittal plane. Image by Thomas Schultz (2006), available at Wikimedia Commons.

Electroencephalography (EEG) and magnetoencephalography (MEG) are two techniques that obtain information about brain behavior from physical principles different from the methods discussed above. Both EEG and MEG measure the activity level of neurons directly by looking how that activity affects electromagnetic fields.

When large groups of neurons fire in a coordinated way, they generate electric and magnetic fields that can be detected. The phenomenon, discovered in animals, was first reported by Richard Caton, a Liverpool physician (Caton 1875). In 1924, Hans Berger, inventor of the technique now known as EEG, recorded the first human electroencephalogram. (See Berger 1929.)

EEG works by recording the brain’s electrical activity over some period of time from multiple electrodes, generally placed on the scalp. The electric potential generated by the activity of an individual neuron is far too small to be picked up by the electrodes. EEG activity, therefore, reflects the contribution of the correlated activity of many neurons with related behavior and similar spatial orientation. Pyramidal neurons in the cortex produce the largest part of a scalp-recorded EEG signal because they are well aligned and because they fire in a highly correlated way. Because electric fields fall off with the square of the distance, activity from deeper structures in the brain is more difficult to detect.

The most common analyses performed on EEG signals are related to clinical detection of dysfunctional behaviors of the brain, which become visible in EEG signals as abnormal patterns. However, EEG is also used in brain research. Its spatial resolution is very poor, since the origin of the signals can’t be located more precisely than within a few centimeters. However, its time resolution is very good (on the order of milliseconds). EEG is sometimes used in combination with another imaging technique, such as MRI or PET.

In a related procedure known as electrocorticography (ECoG) or intracranial EEG (iEEG), electrodes are placed directly on the exposed surface of the brain to record electrical activity from the cerebral cortex. Since it requires removing a part of the skull to expose the brain’s surface, ECoG is not widely used in fundamental brain research with healthy human subjects, although it is extensively used in research with animals.

Magnetoencephalography (MEG), first proposed by David Cohen (1968), also works by detecting the effects in the electromagnetic fields of firing neurons, but in MEG the detectors are very sensitive magnetometers. To reduce the magnetic background noise, Cohen used a magnetically shielded room. (See figure 9.5.)

10998_009_fig_005.jpg

Figure 9.5 A model of the magnetically shielded room built by David Cohen in 1969 at MIT for the first magnetoencephalography experiments. Photo taken at Athinoula A. Martinos Center for Biomedical Imaging.

Whereas EEG detects changes in the electric field caused by ion flows, MEG detects tiny changes in the magnetic field caused by the intra-neuron and extra-neuron currents that occur in the correlated firing of large numbers of neurons (Okada 1983). Since currents must have similar orientations to generate magnetic fields that reinforce one another, it is again (as in EEG) the layer of pyramidal cells, which are situated perpendicular to the cortical surface, that produce the most easily measurable signals. Bundles of neurons with an orientation tangential to the scalp’s surface project significant portions of their magnetic fields outside the head. Because these bundles are typically located in the sulci, MEG is more useful than EEG for measuring neural activity in those regions, whereas EEG is more sensitive to neural activity generated on top of the cortical sulci, near the skull.

MEG and EEG both measure the perturbations in the electromagnetic field caused by currents inside the neurons and across neuron cell walls, but they differ greatly in the technology they use to do so. Whereas EEG uses relatively cheap electrodes placed on the skull, each connected to one amplifier, MEG uses arrays of expensive, highly sensitive magnetic detectors called superconducting quantum interference devices (SQUIDS). An MEG detector costs several million dollars to install and is very expensive to operate. Not only are the sensors very expensive; in addition, they must be heavily shielded from external magnetic fields, and they must be cooled to ultra-low temperatures. Efforts are under way to develop sensitive magnetic detectors that don’t have to be cooled to such low temperatures. New technologies may someday make it possible to place magnetic detectors closer to the skull and to improve the spatial resolution of MEG.

A recently developed method known as event-related optical signal (EROS) uses the fact that changes in brain-tissue activity lead to different light-scattering properties. These changes may be due to volumetric changes associated with movement of ions and water inside the neurons and to changes in ion concentration that change the diffraction index of the water in those neurons (Gratton and Fabiani 2001). Unlike the BOLD effect, these changes in the way light is scattered take place while the neurons are active and are spatially well localized. The spatial resolution of EROS is only slightly inferior to that of MRI, but its temporal resolution is much higher (on the order of 100 milliseconds). However, at present EROS is applicable only to regions of the cortex no more than a few centimeters away from the brain’s surface.

The techniques described in this section have been extensively used to study the behavior of working brains, and to derive maps of activity that provide significant information about brain processes, at the macro level. Whole-brain MRI analysis has enabled researchers to classify the voxels in a working brain into several categories in accordance with the role played by the neurons in each voxel (Fischl et al. 2002; Heckemann et al. 2006). Most studies work with voxels on the order of a cubic millimeter, which is also near the resolution level of existing MRI equipment. Many efforts aimed at making it possible to integrate imaging data from different subjects, and a number of atlases of the human brain have been developed—for example, the Probabilistic Atlas and Reference System for the Human Brain (Mazziotta et al. 2001) and the BigBrain Atlas (developed in the context of the Human Brain Project).

Integration of different techniques, including MRI, EEG, PET, and EROS, may lead to an improvement of the quality of the information retrieved. Such integration is currently a topic of active research.

A recently developed technique that shows a lot of promise to advance our understanding of brain function is optogenetics. In 1979, Francis Crick suggested that major advances in brain sciences would require a method for controlling the behavior of some individual brain cells while leaving the behavior of other brain cells unchanged. Although it is possible to electrically stimulate individual neurons using very thin probes, such a method does not scale to the study of large numbers of neurons. Neither drugs nor electromagnetic signals generated outside the brain are selective enough to target individual neurons, but optogenetics promises to be. Optogenetics uses proteins that behave as light-controlled membrane channels to control the activity of neurons. These proteins were known to exist in unicellular algae, but early in the twenty-first century researchers reported that they could behave as neuron membrane channels (Zemelman et al. 2002; Nagel et al. 2003) and could be used to stimulate or repress the activity of individual neurons or of smalls groups of neurons, both in culture (Boyden et al. 2005) and in live animals (Nagel et al. 2005). By using light to stimulate specific groups of neurons, the effects of the activity of these neurons on the behavior of the organism can be studied in great detail.

Optogenetics uses the technology of genetic engineering to insert into brain cells the genes that correspond to the light-sensitive proteins. Once these proteins are present in the cells of modified model organisms, light of the appropriate frequency can be used to stimulate neurons in a particular region. Furthermore, the activity of these neurons can be controlled very precisely, within milliseconds, by switching on and off the controlling light. Live animals, such as mice or flies, can be instrumented in this way, and the behaviors of their brains can then be controlled in such a way as to further our understanding of brain circuitry.

Optogenetics is not, strictly speaking, an imaging technique, but it can be used to obtain detailed information about brain behavior in live organisms with a resolution that cannot be matched by any existing imaging technique.

Brain Networks

The imaging technologies described in the previous section, together with other information, enabled researchers to study how various regions of the brain are connected, both structurally and functionally. They provide extensive information about the axon bundles that connect regions of the brain and about the levels of activity in various regions when the brain is performing specific tasks.

In studying brain networks, each small region of the brain, defined by a combination of geometrical, physiological, and functional considerations, is made to correspond to one node in a graph. Subjects are then asked either to perform a specific task or to stay in their resting state while the brain is imaged and the level of activity of each of these regions is recorded. Statistical analyses of the data and graph algorithms are then used to study different aspects of the brain’s behavior and structure. When more than one subject is used in a study, the coordinates of the different brains are mapped into a common coordinate system; this is necessary because individual brains differ in their geometry and hence simple, fixed spatial coordinates don’t refer to the same point in two different brains. Linear and non-linear transformations are therefore used to map the coordinates of different brains into the coordinates of one reference brain, which makes it possible to obtain a fairly close correspondence between the equivalent brain regions in different individuals.

As an illustration, let us consider one particular brain network obtained from high-resolution fMRI data. In an experiment conducted collaboratively by researchers from the Martinos Center and Técnico Lisboa, one of the objectives was to identify the strongest functional connections between brain regions. The brains of nine healthy subjects, who were at rest, were imaged on a high-field (7 teslas) whole-body scanner with an isotropic spatial resolution of 1.1 millimeter and a temporal resolution of 2.5 seconds. For different subjects, a number of regions of interest (ROI) were selected, using a method whose purpose was to identify stable and significant ROIs (Dias et al. 2015). The correlations between the levels of activity in these regions were then used to construct a graph, with the edge weights given by the value of the correlations. The graph was processed by identifying the maximum spanning tree, then adding to the tree the strongest remaining connections between ROIs, in decreasing order of their strength, until the average node degree reached 4 (Hagmann et al. 2008). The maximum spanning tree can be found by adapting the method that was used to find the minimum spanning tree, using the algorithm discussed in chapter 4. This procedure yields a network showing 124 regions of interest (figure 9.6). Higher numbers of ROIs lead to graphs that are more informative but are more difficult to interpret visually.

10998_009_fig_006.jpg

Figure 9.6 A brain network obtained by considering the highest correlations among 124 different regions of interest in an average of nine human subjects.

The same fMRI data were also used to study the structure and characteristics of very large graphs derived by considering one ROI for each voxel imaged in each subject. Such graphs have more than a million nodes and up to a billion edges, depending on the minimum level of correlation used to define a connection between ROIs (Leitão et al. 2015).

Hundreds of methodologies have been used to study brain networks in many thousands of studies in diverse fields including psychology, neurology, genetics, and pharmacology. Brain networks can be used to study neural circuits and systems, the firing patterns that characterize specific activities or sensations, brain diseases, and many other features of brain behavior. Olaf Sporns’ book Networks of the Brain covers many of the techniques and objectives of brain network studies and is an excellent reference work on the subject.

Slicing and Dicing

Since the imaging methods described above have limited spatial resolution, fine-grained information about neuron structure and connectivity can more easily be obtained from dead brains, which can be sliced and observed at the microscope in great detail. Once slices of the brain have been obtained, microscopy doesn’t destroy the samples and can provide a spatial resolution sufficient to resolve individual neurons, axons, dendrites, synapses, and other structures.

In conventional microscopy, which is derived directly from the technology used in sixteenth-century microscopes, the entire sample is flooded with light, and thus all parts of the sample in the optical path are excited at the same time. Reflection microscopy and transmission microscopy measure the reflected or transmitted light directly, with a resolution limited by a number of factors (most notably the wavelength of the light used to image the samples). Light visible to the human eye has wavelengths between 400 and 700 nanometers, which means that features much smaller than one micrometer (µm) cannot be resolved. Neuron cells’ bodies range in size from 4 to 50 µm, but synapses are much smaller, filling a gap between neurons that is only 20–40 nm wide (Hubel 1988). Therefore, resolving individual neuron components, such as synapses requires electron microscopy.

Electron microscopy uses electrons instead of photons to image samples. Because the wavelength of an electron can be shorter than the wavelength of a visible-light photon by a factor of 103, an electron microscope has a much higher resolution than an optical microscope. Electron microscopy can resolve features smaller than 0.1 nm, several orders of magnitude smaller than the resolution of optical microscopes. Electron microscopy has, therefore, been used extensively to determine the detailed structures of individual neurons and of small groups of neurons. Electron microscopy, however, suffers from several limitations on its use in biological tissues (it is especially difficult to use in tissues that are kept alive). Other methods can be used to observe living tissues, or tissues stained with biological markers related to specific biological processes.

In fluorescent microscopy, what is imaged is not reflected light but light emitted by a sample that has absorbed light or some other form of electromagnetic radiation. The sample must include in its composition fluorophores—fluorescent chemical compounds that re-emit light upon being excited by visible radiation. The wavelength of the fluorescent light is different from the wavelength of the absorbed radiation, and depends on the nature of the fluorophores. The resulting fluorescence is detected by the microscope’s photo-detector. Since cells can be labeled with different fluorescent molecules related to different biological processes, fluorescent microscopy can be used to trace specific proteins inside cells.

Confocal microscopy (Minsky 1961) uses point illumination and a pinhole close to the sample to eliminate out-of-focus light coming from other parts of the sample. Because only light produced by fluorescence very close to the focal plane can be detected, the image’s optical resolution, particularly in the depth direction, is much better than that of wide-field microscopes.

Multi-photon fluorescent microscopy (MFM) is another technique that has extended the range of application of optical microscopy. Like fluorescent microscopy, MFM uses pulsed long-wavelength light to excite fluorophores within a specimen. In MFM, the fluorophore must absorb the energy of several long-wavelength photons, which must arrive almost at the same time, in order to excite an electron into a higher energy state. When the electron comes back to its ground state, it emits a photon. By controlling the laser sources, it is possible to image live cells in better conditions than the conditions that are possible when using alternative techniques. Two-photon microscopy is a special case of multi-photon microscopy in which exactly two photons of infrared light are absorbed by the fluorophore.

Whatever the microscopy technique used, only very thin slices of brain tissue can be imaged, because it isn’t possible to image sections deep inside the slice. The slices imaged can correspond to slices cut from the surface of the tissue, or to the top layer of a block of tissue.

Slices of brain tissue that have been chemically hardened to make them amenable to slicing can be cut with a microtome or an ultramicrotome. Basically glorified meat slicers, such devices can cut very thin slices of samples. Steel, glass, or diamond blades can be used, depending on the material being sliced and the desired thickness of the slices. Steel and glass blades can be used to prepare sections of animal or plant tissues for light microscopy. Diamond blades are used to prepare thin sections of brain for high-resolution microscopy.

Microtomes have been used for decades to obtain two-dimensional images of brain slices. Researchers seeking to obtain more detailed three-dimensional information about neuron arrangement have taken to imaging stacks of slices and combining the information to obtain complete three-dimensional information. That technique, called serial electron microscopy (SEM), was used to obtain the whole structure of the worm C. elegans in a laborious process that took many years. The difficulty of using SEM lies in the fact that, as slices are removed and put into glass blades, they become distorted and difficult to align. Despite these difficulties, SEM has been extensively used to obtain information about three-dimensional brain structures.

A significant improvement over serial electron microscopy came when Winfried Denk and Heinz Horstmann (2004) proposed serial block-face electron microscopy (SBEM). In SBEM, the face of a whole block of tissue is imaged by putting the block inside an electron microscope. Then a thin slice of the block is removed with a diamond blade microtome, and is discarded. The newly exposed slice is then imaged again. The process can be automated and obtains images with less distortion than does SEM.

Images obtained by SEM and by SBEM can be processed to reveal the three-dimensional geometry of individual neurons and even the locations of individual synapses. This is a process essentially similar to computed tomography. Together, two-dimensional imaging and computerized combination of images have led to many different techniques now used in various circumstances and settings. Many different combinations of microscopy techniques, tissue preparation, serial slice analysis, and computerized image processing have been used by researchers to obtain very detailed information about larger and larger brain structures.

Conceptually, the whole structure of connections in a whole brain can be mapped, even though the state of the art still imposes significant limitations on what can be done. A number of efforts to map large regions of mammalian brains have been undertaken by researchers, and the data they have obtained have been made available in the Open Connectome Project.

In 2011, researchers from Harvard University and Carnegie Mellon University used two-photon calcium imaging (a technique used to monitor the activity of neurons in live brain tissue) and serial electron microscopy to obtain both functional and structural information about a volume of the mouse visual cortex measuring 450 × 350 × 52 µm and containing approximately 1,500 neurons (Bock et al. 2011). The researchers used two-photon calcium imaging to locate thirteen neurons that had responded to a particular stimulus, then reconstructed a graph of connections of those thirteen neurons.

In 2013, a group of researchers reported the reconstruction of 950 neurons and their mutual contacts in one section of the retina of a mouse (Helmstaedter et al. 2013). They used serial block-face electron microscopy to acquire sections from an 114-by-80-µm area of the mouse retina. The data were annotated by human curators and by machine learning algorithms to yield the full structure of that region of the mouse retina. Helmstaedter et al. estimated that more than 20,000 annotator hours were spent on the whole process.

In work reported in 2015,, researchers from Harvard, MIT, Duke University, and Johns Hopkins University fully reconstructed all the neuron sections and many sub-cellular objects (including synapses and synapse vesicles) in 1,500 cubic micrometers (just a little more than a millionth of a cubic millimeter) of mouse neocortex, using 3 × 3 × 30 nm voxels (Kasthuri et al. 2015). They obtained 2,250 brain slices, each roughly 30 nm thick, using a tape-collecting ultramicrotome equipped with a diamond blade. The slices were imaged by serial electron microscopy; then the images were processed in order to reconstruct a number of volumes. In these volumes, the authors reconstructed the three-dimensional structure of roughly 1,500 µm3 of neural tissue, which included hundreds of dendrites, more than 1,400 neuron axons, and 1,700 synapses (an average of about one synapse per cubic micron). Figure 9.7, reprinted from one of the articles that describe this effort, shows a rendering of one small cylinder of neocortex.

10998_009_fig_007.jpg

Figure 9.7 A rendering of one cylindrical volume of neocortex, roughly 8 µm in diameter and 20 µm long, obtained from the reconstruction performed by Kasthuri et al. (2015). Reprinted with permission from Narayanan Kasthuri.

The efforts described above and many other efforts have shown that it is possible to obtain highly detailed information about fine-grained brain structure. Such information may, in time, be sufficiently detailed and reliable to define the parameters required to perform accurate neuron-level simulation of brain sections.

Brain Simulation

As our understanding of the mechanisms and structures of living brains evolves, it may become possible to use engineering approaches to replicate and to study in great detail several different phenomena in the brain. When the fine structure of brain tissue and the mechanisms that control brain development are understood in detail, it will become possible to simulate brain development and behavior. The simulations will have to reproduce the complex biochemical mechanisms that direct neuron growth, the chemical and electrical basis for synapse plasticity (and other forms of brain plasticity), and, eventually, the detailed workings of the billions of neurons and trillions of synapses that constitute a working brain. However, if good models exist for these components, there is no fundamental reason why simulation of large regions of a working brain cannot be performed.

It is interesting to assess what the simulation of a large part of the brain, or even the emulation of a whole brain, would imply in computational terms. One of the most ambitious such effort to have been completed so far probably is the Blue Brain Project, a collaboration between the École Polytechnique Fédérale de Lausanne and IBM. The project started in 2006, and the initial goal was to simulate in a supercomputer one neocortical column of a rat, which can be viewed as the smallest functional unit of the mammalian neocortex. A neocortical column is about 2 mm tall, has a diameter of 0.5 mm, and contains about 60,000 neurons in a human and 10,000 neurons in a rat. A human cortex may have as many as 2 million such columns. The computing power needed to perform that task was considerable, as each simulated neuron requires computing power roughly equivalent to that of a laptop computer. However, supercomputing technology is rapidly approaching a point at which simulating large parts of the brain will become possible.

In the most recent work, the results of which were published in 2013, the model and the simulations used the software and hardware infrastructure of the Blue Brain Facility to model a cortical column of a rat, with 12,000 neurons and 5 million dendritic and somatic compartments. The simulations were performed using publicly available software, the NEURON package, running on a Blue Gene P supercomputer with 1,024 nodes and 4,096 CPUs (Reimann et al. 2013). Since so far we have no detailed structural data on the cortical columns of a rat, statistical information about neuron connectivity and synapse distribution was used to define the parameters of the model. We can extrapolate the results of that simulation to calculate the amount of computer power required to simulate more complex systems and even a complete brain. Of course, there is no conclusive evidence that the level of detail used to simulate this “small” subsystem of a rat’s brain is adequate for more complex and powerful simulations, even though Reimann et al. argued that a behavior consistent with reality was observed in this particular simulation. However, let us assume, for the moment, that this level of detail is indeed appropriate and could be used to simulate accurately more complex systems.

The Blue Gene P supercomputer, in the configuration used in the aforementioned study, has been rated as able to perform about 14 teraFLOPS (trillion floating-point operations per second). That speed enabled the computer to simulate about 4 seconds of cortical activity in 3 hours of computer time, in effect simulating the system approximately 2,700 times slower than real time. The fastest computer available at the time of this writing— the Sunway TaihuLight supercomputer, with 10 million processor cores—has been rated at 93,000 teraFLOPS (93 petaFLOPS), about 6,600 times the speed of the Blue Gene P supercomputer used in the simulation mentioned above.

If we ignore problems related to the difficulty of scaling up the simulation to a computer with a much larger number of cores (problems that are real but can be addressed), then, in principle, the Sunway TaihuLight supercomputer could simulate, more than two times faster than real time, a 12,000-neuron cortical column of a rat. With all other things assumed equal (even though we know they probably are not), if we had a complete model of a human brain such a machine could simulate its 86 billion cells at a speed roughly 3 million times slower than real time. To simulate 10 seconds of real brain time (enough to simulate, say, the utterance of one small sentence) would require about one year of computer simulation. Even more striking is the difference in the amounts of energy needed by the supercomputer and by an actual brain. A brain, which uses about 20 watts of power (Drubach 2000), would spend about 200 joules to work for 10 seconds, whereas the 15MW Sunway TaihuLight would spend roughly 5 × 1014 joules to simulate the same event during this year, consuming 2 trillion times as much energy as the brain would.

The technologies involved, however, are evolving rapidly, making the simulations more and more efficient. The most ambitious simulation of a network of spiking neurons performed to date, developed to test the limits of the technology (Kunkel et al., 2014), used a supercomputer to emulate a network of 1.8 billion neurons, with an average number of 6000 synapses per neuron. The model used is not as detailed as the model used in the Blue Brain project, leading to a faster simulation at the expense of some precision.

The simulation used 40 minutes of the 8 petaFLOPS K computer, the fourth-fastest computer in the world at the time, to model the behavior of one second or real time of this large network of spiking neurons. Extrapolating to the size of a whole brain, and assuming no negative impact from the scaling up, the simulation of all the neurons in a human brain would still proceed 100,000 times slower than real time. The Sunway TaihuLight would have performed such a simulation “only” 10,000 times slower than real time, if all other parameters remained comparable.

The first reaction to the numbers cited above may be a combination of surprise, despair, and relief based on the renewed understanding—and now on comparatively hard data—that the human brain is simply too complex, too powerful, and too efficient to be fully simulated in a computer. In fact, even if we could obtain a working model of a brain (far in the future), and even if the fastest computer on Earth, allocated only to that task, could simulate such a model, the simulation would proceed many times slower than real time. This simple reasoning seems to show that such an endeavor will be forever impossible. However, because of two factors that cannot be ignored, that apparently obvious and clear conclusion is too naive.

The first of those two factors is the exponential growth in the power of computers, which hasn’t stopped and which is likely to continue for quite a while. If Moore’s Law continues to define the evolution of computers, in practice doubling their performance every two years, and if that rate of evolution is also true for the largest supercomputers, then a computer with the power to perform real-time simulation of a human brain will be available around the year 2055. There are a number of assumptions in these computations that may well prove to be false, including the sustainability of the evolution predicted by Moore’s Law. Nonetheless, it remains likely that such a computer will exist in 40 or 50 years.

The second factor is, in my opinion, even more important than the first. It was noted in chapter 7 that simulation of digital chips is performed using multi-level simulation in which details of the behaviors of devices are abstracted away at the higher levels of the simulation. This means that systems with billions of transistors can be effectively simulated with multi-level models of the detailed workings of each individual transistor. The utilization of multi-level simulation techniques increases the speed of simulation by many orders of magnitude. There is no essential reason why such an approach cannot be effectively applied to the simulation of a complete brain. In fact, a number of authors—among them Marasco, Limongiello, and Migliore (2013)—have proposed ways to accelerate the simulation of neurons that would result in significant speedups. It is true it is much easier to create a multi-level model of an integrated circuit, with its well-defined modules and well-understood structure, than to create a multi-level model of a brain, with its complex network of connections and poorly defined modules. I believe, however, that the majority of the advances in this area are still to come. As more and more effort is put into modeling the behaviors of parts of the brain, more and more resources will be needed to run simulations. Multi-level abstractions of more complex brain blocks will be created and will be used to create higher-level models of brain regions—models that can be simulated much more rapidly in a computer without sacrificing any of the precision that is essential for the accurate simulation of a working brain.

Armed with a better understanding of computers, biological systems, and brains, let us now proceed to the central question that is the main topic of this book: Will we be able to create digital minds?