Reductionism in Art and Brain Science: Bridging the Two Cultures

HOW THE BRAIN PROCESSES AND PERCEIVES ABSTRACT IMAGES

The reason that figurative art, such as portraiture, can exert such a profound impact on us is that the visual system of our brain has powerful bottom-up machinery for processing scenes, objects, and particularly faces and facial expressions. Moreover, we respond more strongly to the exaggerated depiction of a face by an Expressionist artist such as Oskar Kokoschka or Egon Schiele because the face cells in our brain are tuned to respond more powerfully to exaggerated facial features than to realistic ones.

How, then, do we respond to abstract art? What machinery in the brain enables us to process and perceive paintings whose images have been radically reduced, if not eliminated? One point that emerges clearly is that many forms of abstract art isolate color, line, form, and light, thus making us implicitly more aware of the functioning of the individual components of the visual pathway.

The idea that has driven much of modern research on the perception of art came from Ernst Kris’s insight that each of us perceives a given work of art somewhat differently, so that viewing a work of art involves a creative process on the part of the beholder. As we saw in chapter 3, Ernst Gombrich applied Kris’s ideas about ambiguity in art to the entire visual world by focusing on the inverse optics problem.

Each of us takes incomplete information from the outside world and makes it complete in our own distinctive way. The reason we succeed in reconstructing a three-dimensional image from reflected light—and in doing so accurately most of the time—is that our brain provides context not only through bottom-up but also through top-down processing of visual information. As we have seen, bottom-up information is supplied by the computational logic built into the circuitry of the visual system, such as the ability to recognize faces, while top-down information is supplied by cognitive processes such as expectation, attention, and learned associations.

Thomas Albright and Charles Gilbert have recently made substantial progress in understanding top-down processes, especially the learning mechanisms that underlie them. In discussing these processes we must first distinguish between sensation, which is more closely related to bottom-up processing, and perception, which is more closely related to top-down processing.

SENSATION AND PERCEPTION

Sensation is the immediate biological consequence of stimulating a sensory organ, such as the photoreceptors in our eyes. Sensory events can affect our behavior directly, but they lack context. Perception, as we have seen, incorporates the information our brain receives from the external world with knowledge based on learning from earlier experiences and hypothesis testing. Thus, as it pertains to vision, perception is the process whereby reflected light becomes linked to an image in the environment, is made enduring by the brain, and becomes coherent when the brain assigns it meaning, utility, and value.

A critical element in perception is the recruitment of associations between a particular sensory event and other images and other sources of information. These associations provide the context necessary to resolve the ambiguity inherent in all sensations (Albright 2015) and, as Kris noted, the ambiguity inherent in a work of art. The American philosopher William James draws the distinction between sensation and perception in this way: perception differs from sensation by recruiting the consciousness of further facts associated with the object causing the sensation (James 1890).

8.1 The central problem of vision is twofold: one component is optical and the other perceptual. The optical problem—the problem of sensation—involves the reflection of light off surfaces in the visual environment, resulting in an image on the retina. The perceptual problem involves identification of the elements of the visual scene that gave rise to the retinal image. This is a classic inverse problem for which there is no unique solution: a given retinal image could be caused by any one of an infinite set of visual scenes.

The distinction between sensation and perception is the central problem of vision (Albright 2015). Whereas sensation is optical and involves the eyes, perception is integrative and involves the rest of the brain as well (fig. 8.1).

As we have seen, learning and memory include the strengthening of specific synaptic connections in the brain. Albright and Gilbert discovered that top-down processing is the result of a critical computation in which brain cells use contextual information to transform incoming sensory information, such as that from a work of art, into an internal representation, or percept (Albright 2015; Gilbert 2013b).

Where in the brain does the synaptic strengthening that contributes to this top-down processing occur? A significant body of evidence now suggests that one key site for the storage of long-term memories of associations is the inferior temporal cortex, a region with direct connections to the hippocampus (fig. 8.2), where explicit memories of people, places, and objects are encoded.

The inferior temporal cortex represents the pinnacle of the brain’s visual information-processing hierarchy and is known to be important for object recognition, which relies on our memory of prior associations. Bottom-up signals sent by sensory neurons in response to objects in our visual field are processed into representations of those objects in the inferior temporal cortex, where, for example, the face patches are located. Learning to associate one object with another is achieved by strengthening the connections between the neurons representing each object; this is done via the indirect pathway (fig. 8.2). The resulting association is consolidated in the inferior temporal cortex and stored by memory structures in the medial temporal lobe.

Albright and his colleagues tested this idea by training monkeys to associate pairs of visual stimuli that consisted of meaningless patterns. As each monkey was learning the association, the scientists monitored the activity of neurons in the monkey’s inferior temporal cortex. They found that at the outset, the neurons responded selectively to each object, each visual pattern. As the monkeys began to learn to associate two stimuli, the connections between neurons that initially responded to one or the other visual pattern were strengthened. These changes in the neurons are physical manifestations of classical conditioning, of newly learned association. Memories consolidated in this manner, over the course of a lifetime, provide us with further information that we then associate with a perceived object through the indirect pathway.

The indirect pathway can also be activated by the contents of working memory—that is, by feedback from the prefrontal cortex, which is concerned both with aspects of working memory and with executive functions. Under normal conditions a visual experience results from combined direct and indirect inputs to the inferior temporal cortex. However, we have reason to believe that the changes in neural connections underlying learned associations are not limited to inputs to the inferior temporal cortex: they reflect a general capability of the entire visual system—in fact, of all sensory systems. Evidence for this idea comes in part from imaging studies of brain activity during learning, which have revealed changes in neuronal activity even in regions where the very early stages of visual processing take place.

8.2 Circuits for visual association and recall. Bottom-up signals from sensory cells lead to representations of a circus tent and a horse in the inferior temporal cortex. Before associative learning, a neuron (in light blue) responds well to the circus tent but not to the horse. Learned associations between the two objects are consolidated in the inferior temporal cortex by strengthening connections between the neurons representing each of the paired objects via the indirect pathway. Thus, activating the indirect pathway results in recall of the circus tent following presentation of the horse.

This finding led Albright to explore associative learning by examining the response properties of neurons in the medial temporal cortex, which occupies an intermediate position in visual processing. And, indeed, he found changes in connectivity that are similar to the changes seen in neurons in the inferior temporal cortex.

Alumit Ishai and her colleagues have illustrated this point further (Mechelli et al. 2004; Fairhall and Ishai 2007). When they showed study volunteers actual images of faces or houses, the images activated areas of the visual cortex engaged in the early stages of processing. In contrast, when they asked the volunteers to recall the images of faces or houses, two regions of the brain involved in top-down processing were activated: the prefrontal cortex and the superior parietal cortex. The prefrontal cortex responds only to figurative images that the brain can fit into a known category, such as a face or house or cat—images that convey content. The superior parietal cortex, which manipulates and rearranges information in working memory, is activated by any visual image.

These results indicate that sensory representation of faces and objects in the brain is largely mediated by bottom-up processes arising in the early visual areas, whereas perception of images derived from memory storage is mediated in good part by top-down mechanisms originating in the prefrontal cortex (Mechelli et al. 2004).

Thus, when we look at a work of art, information from several sources interacts with the incoming pattern of light to yield our perceptual experience of that work. Much information is conveyed to the brain from bottom-up processing, but important information is also added from our memory of prior encounters with the visual world. These memories of other experiences with other works of art enable us to infer the cause, category, meaning, utility, and value of the images on our retina.

In sum, the reason we can usually resolve the ambiguity of a retinal image accurately because our brain supplies context. Context, broadly speaking, consists of other bits of information: those that are also present in the retinal image; those, such as face processing, that are inherent in the computational machinery of the brain; and finally, those learned from prior experience with the world, including the world of art.

As early as 1644 René Descartes argued that visual signals originating in the eye and those originating from memory are both experienced by the conversion of information onto a common brain structure. This idea has recently been supported by functional brain imaging studies, in which people are asked to imagine specific visual stimuli or are trained to recall an image through association with another image. The studies have documented patterns of activity in a variety of brain regions involved in low- and intermediate-level visual processing.

Thus, what abstract artists contend—and abstract art itself bears out—is that an impression, a sensory stimulation of the retina, is merely a spark for associative recall. The abstract painter does not attempt to provide pictorial detail, but rather to create conditions that enable the viewer to complete the picture based on his or her own unique experience. Legend has it that upon viewing a sunset painted by Turner, a young woman remarked, “I never saw a sunset like that, Mr. Turner,” to which Turner replied, “Don’t you wish you could, madam?”

The pleasure that many viewers derive from abstract art is an example of what James terms “the victorious assimilation of the new”—the coherent perceptual experience of something we have never quite seen before—by its association with familiar things (James 1890). I would argue further that the assimilation of the new—the recruitment of top-down processes as part of the beholder’s creative reconstruction of the image—is inherently pleasurable because it stimulates our creative selves and contributes to the positive experience that many beholders have in the presence of certain works of abstract art.

REVISITING THE ABSTRACT PAINTING OF DE KOONING AND POLLOCK

A simple example of the importance of the associations we bring to bear on abstract art is illustrated in figures 8.3 and 8.4. Unless we have seen figure 8.4 before, we have difficulty grasping what this apparently random pattern of light and dark regions symbolizes. But our perceptual experience is radically altered when we view figure 8.4, which provides enough information to resolve the ambiguity in the earlier image. Moreover, after we have seen figure 8.4, our next encounter with figure 8.3 will be markedly different because of the information we bring to bear on it from memory. In fact, our perception of the original image may have been permanently altered.

Ramachandran and his colleagues examined the cellular mechanisms underlying this phenomenon (Tovee et al. 1996). They found that the response of single neurons in the inferior and in the medial temporal cortex can change rapidly, showing the effects of learning after as little as five to ten seconds of exposure to the unambiguous image. After this brief exposure, the brain applies that learning to the ambiguous image.

These cellular experiments in monkeys and parallel psychophysical studies in humans illustrate that people and higher primates are capable of rapid learning about stimuli in the visual world. This may be why we can recognize faces and objects that we have seen for only a few seconds. These findings are also consistent with Albright’s and Ishai’s discovery that neurons in the inferior and medial temporal cortex are part of a top-down processing system that includes the prefrontal cortex and the superior parietal cortex.

With this example of associations in mind, let us reexamine de Kooning’s and Pollock’s transition from figuration to abstraction.

De Kooning’s Seated Woman, painted in 1940 (fig. 8.5), is a depiction of Elaine Fried, whom he would marry three years later. This was one of de Kooning’s first paintings of a woman, and it already contains some fascinating abstract touches. For example, her right eye, the right side of her face, and her right arm are less clearly defined than the left. This demands some top-down processing on our part. Why is de Kooning disassembling her body? The only parts that appear intact and symmetrical are her breasts. This is an early attempt by de Kooning to redefine the female form.

8.3 To most observers, this image initially appears as a random pattern with no figure clearly visible. The perceptual experience elicited by this stimulus is radically, and perhaps permanently, altered after viewing the pattern shown in figure 8.4 (adapted from Albright 2012).

Excavation (fig. 8.6), done ten years later, is a very different work that is largely abstract. It is essentially flat, with little sense of perspective. But knowing de Kooning and his fascination with the female form, our imagination need not wander far on this canvas to encounter rounded shapes that, depending on one’s mood or disposition, easily recall one or more women alone or coupling with another person. There are almost unlimited possibilities in this painting for both bottom-up resolution of ambiguity and top-down imagination, as we can recall one possible association after another. The contrast between Seated Woman and Excavation is all the more remarkable because Seated Woman is already quite ambiguous.

8.4 This image demonstrates the influence of associative pictorial recall (top-down signaling) on the interpretation of a retinal stimulus (bottom-up signaling). Most observers experience a clear, meaningful percept upon viewing this pattern. After seeing this image, the interpretation of the pattern in the previous figure (8.3) is perceived quite differently. Having once perceived figure 8.4, the observer can produce a figural interpretation driven largely by images drawn from memory (adapted from Albright 2012).

8.5 Willem de Kooning, Seated Woman, 1940

Now let us turn to Pollock’s early figurative painting Going West (fig. 8.7). We are struck by the ceaseless clockwise movement, beginning with the mules and their drivers, moving to the clouds and the moon in the sky, and coming around to the mules again.

8.6 Willem de Kooning, Excavation, 1950

Fifteen years later, in Number 32 (fig. 8.8), Pollock again creates a powerful sense of movement. But now, rather than forcing a single, compelling clockwise movement, Pollock frees us to choose any direction—or, more likely, several directions—to track, as our eyes scan the canvas seeking order. Are there any images here? Is any direction of movement particularly dominant? The painting calls to my mind a battle of images, a scene of never-ending image warfare. It takes our breath away if we let our imagination wander to other scenes that might in any way be comparable.

8.7 Jackson Pollock, Going West, 1934–35

8.8 Jackson Pollock, Number 32, 1950

What stands out in this comparison of de Kooning’s and Pollock’s paintings is that while abstract art is clearly reductionist in terms of figuration, it can encourage our imaginative capabilities to a far greater degree than many figurative counterparts would. Moreover, the two fully abstract paintings are less challenging to the bottom-up processing by our brain’s visual machinery than Cubist art is. Cubist art often retains figurative elements, yet it asks us to explore them from different, unrelated perspectives that our brain did not evolve to process meaningfully. The abstract paintings, in contrast, seem to rely less on bottom-up processing, which primarily serves to resolve potential ambiguities; instead, they rely heavily on our imagination, our top-down associations from personal experiences and encounters with other works of art.