12 Correspondences: Uniting Image and Sound
Ernest Edmonds
12.1 Introduction
Chapter 9 discussed the significance of code, computer programs, in much current art. The emphasis was on the interest that many artists have in structure and ordered systems underpinning the construction of their artworks and the revolutionary opportunity that the writing of computer code offered in this respect. Indeed, the development of interest in formal structures was also the basis of the historical developments, pre- and postcomputer, discussed in chapter 10.
Beyond structure and order, the very nature of the abstract symbol manipulation that is at the heart of code has another significant implication. To relate the symbols to reality, mappings are required. The mappings may relate a detected sound or movement, for example, to a particular symbol. Equally, they may relate symbols to particular colors or tones, and so on. Thus, two of the key elements that define a computer-based artwork will often be the code and the relevant set of mappings.
In time-based and performance works the code and the mappings may change over time. In live coding the code itself is changed during the performance. Equally, the performers can change the mappings as the piece develops.
There has been an interest in artworks that combine image and sound in various ways from well before the invention of the computer, as we see later. The computer-based notion of code and mappings, however, introduces a new possibility for the realization of such art. The code itself is essentially media independent. Only when the mappings are applied does any particular physical medium come into play. Most particularly, the same code can be mapped onto both image and sound, for example. In fact, specific aspects of the code, particular symbols, might be mapped to sound and others might be mapped to image elements, such as color. This possibility has enabled the developments discussed in this chapter.
In art and entertainment it is no longer possible to consider the purely visual representative of the norm. The integration of music or other sound elements has become common. This chapter reviews these issues in the specific context of audiovisual art and its recent manifestation in digital works, although a historical, precomputer, review is also presented so as to put this recent work in its long-term context.
12.2 Historical Context
Many people used to think a clear distinction between the visual arts and music composition exists: a painter is not a composer, a sculptor is not a musician, brushes and chisels are not violins or pianos, and colors are not sounds. We can also find an immense amount of literature for both music and visual art history that follow, and therefore stress, this distinction.
On the basis of a clear distinction between the art of the eye and the art of the ear, one cannot classify the increasing amount of art works and instruments that stimulate, couple, and interconnect sounds and colors. Questions then start to arise. When in history did the need for combining sounds and colors come about? What are the predecessors of cinema, television, and computer? It seems plausible to be able to outline a history, parallel to that of music and visual art, that traces the evolution of image-sound systems, or audiovisual systems, and the corresponding art works. It is a difficult task to place the main phases of this history, mainly because to do so requires the identification in visual art history and music history of the connecting points, the history of the in-between. Divisions blur, and the historical classification of works as purely musical or purely visual is questioned. How important, for the whole perception of an acoustical musical work, is it to watch the musicians playing in concert? Is the experience of hearing musical patterns and rhythms an integral part of the perception of patterns of colors and shapes in space? How much does the eye influence the ear and vice versa?
Since antiquity, thinkers and philosophers have searched for unifying principles that could explain our multisensory experience of the world. Such an all-embracing harmony principle was often believed to be based on numbers. The search for a scientific relationship between color and sound can be considered as a part of this wider search for harmony. Pythagoras discovered the relationship between musical sounds, the length of strings, and the division in octaves (van Campen 1999); Aristotle produced a color theory in which he related the consonant quality of tone intervals to colors (Jewansky 1999). In the sixteenth and seventeenth centuries artists like Arcimboldo (Gage 1993) and thinkers such as André Félibien, historiographer and architect (Jewansky 1999), produced scale systems and theories about the relationship between colors and sounds.
With Newton’s Opticks (1704), the physical relationship between color and sound, as we know it today, was revealed. From that moment, it was understood that both colors and sounds could be described in terms of wavelengths, bandwidths, and frequencies. In the eighteenth and nineteenth centuries, instrument makers started to build machines or instruments that could stimulate simultaneously both the aural and the visual senses. They are the first examples of machines with which it was possible to create an abstract audiovisual composition in real time. The relationship between the colors and the notes was predetermined by the builder of the instrument and based on mathematical or perceptual systems. These instruments were often called color organs, and summaries of their history can be found in papers by Kenneth Peacock (1991a, 1991b). Color organs were built by Louis Bertrand-Castel (clavecin oculaire), D. D. Jameson, Bainbridge Bishop, A. Wallace Rimington, Frederick Castner (pyrophone) and Thomas Wilfred (clavilux).
The term “colour organ” was first used in a patent application by Rimington in 1893 (Peacock 1988). These instruments often looked like typical musical instruments but when played they controlled colored gas lamps or colored paper strips illuminated by candles, for example. The production of these instruments stimulated discussions regarding the possible analogy between color and sound. Similarities and differences were noticed and discussed. Examples of subjects of these discussions are the different concept of consonance in sound and in color (consonance in sound is due to physical properties of sound, but it is culturally based for colors); dissonance in music is often thought to be more unpleasant than dissonance in color; two colors create a new unit, but two tones do not create a new tone; and a series of colors cannot be memorized as easily as a series of tones that build up a melody (Jewansky 1999).
12.3 An Audiovisual Discourse
The majority of people make associations between colors and sounds, but these associations have historical and cultural reasons, not physical ones. Thus, there has been a considerable interest in finding or creating systems of correspondences and instruments that could allow the manipulation of those correspondences, one that treats the audio and visual realms almost as a whole. At the end of the nineteenth century, in many ways visual arts were acquiring characteristics typical of music. These were mainly the use of colors in an abstract way (sounds have normally been abstract) and the introduction of the sense of movement and therefore time.
This can be seen in the development of abstract art; in Futurism, in which the representation of movement and speed was central; Calder’s mobiles; and Fluxus, in which the integration of paint, music, poetry, and performance was fundamental. At the same time aspects typical of the visual arts acquired increasing importance in music. In the twentieth century the simultaneous manipulation of sound and color saw a great expansion due to the technological applications of two consecutive discoveries: electricity and the conversion of analogue electric signals into digital information. These two factors allowed the inventions of new technologies. Recording visual moving images and sound was a revolution that influenced music and art making forever. Siegfried Zielinski, in the 1999 book Audiovisions, writes, “My conceptual starting point is: over the last 150 years, in the history of the industrially advanced countries, a specialized, tending to become ever more standardized, institutionalized area of expression and activity has become established. I call it the audiovisual discourse. It encompasses the entire range of praxes in which, with the aid of technical systems and artifacts, the illusion of the perception of movements—as a rule, accompanied by sound—is planned, produced, commented on, and appreciated” (Zielinski 1999, 18).
In the middle of the twentieth century, score writing became a visual outcome in itself, as was well shown in the exhibition Eye Music promoted in 1986 by the British Arts Council (Drew and Harrison 1986), and the movement of sound in space, possible with the new playback technology, became an important parameter of music production as pitch and rhythm always were. The use of recorded or synthesized sound samples in music pushed composers to investigate two important elements of sound that are closely linked to the visual realm: one is timbre, or tone color, and the second is the effect on music perception of the absence from the audience’s view of the sources of the sounds. This second factor showed how much the visual aspect of a performance (for example, watching an orchestra playing in concert) is part of the musical experience itself. That is a possible reason much of today’s digital music is performed in public together with a digital visual show.
When we watch a film we accept film music as a natural part of the work. More generally, the sound track is recognized as a crucial element in the quality of the film in its total sense. At times the music is thought of as an accompaniment to the visual element, at others it might alternatively be thought of as having equal weight and importance. Much the most interesting integration, however, is when the music and the visual element are equal so that, for example, one can see a visual display as one instrument in a piece in which other instruments, such as violins, happen to produce sound. The composition of such work can begin either with the music or the visual or swap between them. Alternatively, it might begin from some more abstract description or notation that can be mapped onto either sound or image.
Many have been inspired by the idea of integrating sound and vision. by Baudelaire’s poem “Correspondences,” in which he speaks of “long-held echoes, blended somewhere else into deep and shadowy unison … the sounds, the scents, the colors correspond” (Baudelaire 1982). There is considerable interest in this integration today, and as I have argued, the advent of the computer has enhanced the possibilities in this area enormously. There are two key reasons for this. One is its ability to control the real-time production of sound and image with considerable flexibility and speed. The second reason for the importance of the computer is that it is basically a symbol-manipulation machine, and so it is able to take a symbolic form (such as a score, which is a symbolic representation of music) and automatically work with it by, for example, transforming it according to a rule. We can operate with structures that mediate between sound and vision so that a unified work can be produced that integrates both. Thus, a single unified abstract structure is mapped onto sound and image to produce the integrated synesthetic work.
12.4 Live Coding
A quite different kind of correspondence is one that exists between the music, for example, and the code. As mentioned in chapter 10 there is a significant movement that performs live coding (Brown 2006). The performers write and modify code to generate the music during the performance itself. The distinguishing feature of this kind of work is that the code is shown to the audience, usually by displaying it in prominent projections. So from the audience’s point of view, the performance is audio and visual.
Live coders take the view that, because the code has a key role in their art, it is part of the art. Of course, one may argue with this position, but it clearly has merit, particularly in the live situation when the code is being changed to generate different results. So here we have a special case of the audiovisual experience. For the most part we do not find live coders paying as much attention to the visual experience relating to seeing the code as they do to the audio experience of hearing the music. So the visual part is more cerebral than the sound. It is an important line of work in that it brings the code to the foreground in a way that no other current artistic practice does. Its future may be in developing that aspect of the work into a true aesthetic experience equal to the music that the live coders make.
12.5 Correspondences: A Personal Exploration
In my own work, the idea is to operate with abstract structures that unify sound and vision. As well as fully composed pieces, it is possible to introduce improvisation or other performance attributes. For example, the sounds made by a musician can be analyzed in real time so that an extra note or a change in rhythm can be fed into the image-generation system and influence its development. In this way, an improvised duet between a sound instrument and an image instrument becomes possible.
My time-based work uses generative procedures that relate closely to those often used in music today (Edmonds 2003). In a series of collaborations through the 1990s and beyond I have worked with composers to make audiovisual abstract performance pieces. The underlying concepts for all these works is that a single structural form, represented in the code, generates both audio and visual representations. These audio and visual representations are not necessarily equivalent—that is to say, they need not have a one-to-one mapping, but they are part of the same underlying generative structure. Typically, a generative system is implemented in Max/MSP that produces a sequence of vectors according to a particular set of rules. The programs incorporate two sets of parallel mappings from these vectors to image and sound data. The implicit correspondences between audio and visual information are not intended to correspond to any particular physical or psychological theory. Rather, they constitute the specific aesthetic of the given work. In both domains the style is very minimal, and the works can be seen in part as developments from the Systems art tradition (Bann 1972). For example, the visual element might consist of a changing display of between one and eight stripes of closely related colors. In one case there are just two stripes, and the saturation gradually increases during the piece. The hues are selected by the generative system, and the brightness of each stripe is under the direct control of a different performer. The audio parameters are treated in a parallel manner.
I worked with Jean-Pierre Husquinet and a group of Belgian musicians on pieces in which the live music was combined with a video construct. The first such performance was in Liège in 1990, and others followed over the next few years. In these works each musician was allocated one component of the image to follow as a score, and a set of rules provided for a direct reading of the image into music. In 2000 I worked with another composer, Maddy Aldis, on a work that was performed that year at the opening of my exhibition at Loughborough University. In this piece, Correspondence 1, a sequence of patterns, in the abstract, was first defined. Correspondences were determined between those patterns and, on the one hand, projected images and, on the other, a score for violin. In 2003 I began working with Mark Fell on purely electronic works. As with Correspondence 1, a pattern sequence was mapped onto both visual and audio realizations but, in these later works, both the audio and the visual streams were generated completely from code. The formal structures used are medium independent and are used together with sets of mappings onto the visual and the audio domains. The audiovisual works with Mark Fell developed into a number of forms. These were performances, a film,1 and installation works that went on changing over long periods and were shown in gallery settings.2
The work Kyoto to Sheffield, for example, is displayed on a wall-mounted plasma screen, with sound relayed over a small pair of audio monitors or headphones. It is a generative piece written using Max/MSP/Jitter software.
The software consists of several discrete processes. These are grouped into three sections: (1) pattern generation, (2) image display, and (3) a representation of the mappings used between the symbols manipulated in section 1 and the physical sound and image events.
DC_Release is another generative audiovisual performance work composed with Mark Fell. It is performed from two Apple laptops running Max/MSP and related software. The work is part of a series that began in 2003. The core concept is that of an integrated audiovisual experience. The work is driven from an internal generative structure from which audio and visual mappings are used to realize the physical manifestation of the piece. The colors and sounds are drawn from carefully selected palettes that define the general nature of the piece. The generative processes, which control selection, sequencing, and timing, are modified in real time by the performers. The time base and the shifting relationships between performer and generative (automatic) control define the central shape of the piece. DC_Release begins with slow, simple performer-controlled changes and progresses to exhibit increasing generative control and a cumulative quality. The first performance was in the Corcoran Gallery in Washington, D.C., April 2007, as part of the ColorField Remix festival. Subsequently, it was performed in Sydney, Australia, and in the UK in London, Sheffield, York, and Leeds.
The use of digital computers creates changes in the established work practices, opening new possibilities to explore. For example, the digital nonlinear editing now possible marks the transition from a horizontal to a vertical method of working in video and cinema. Now, with computers, every operation is become a kind of expanded data-processing operation. Computers are the most extendible and all-comprehensive machines that we produce at this moment in history. Using computers, any type of audiovisual composition can be created: descriptive or abstract, interactive or noninteractive, with direct mathematical correspondences between sound and visual parameters or with an intuitive, metaphorical relationship between sounds and images, and so on.
Computers are the instruments with which the simultaneous manipulation of sound and images can be done at a very deep level, creating cross- or intermedia art systems. All the information, whether visual, auditory, or textual, is represented in the same digital code. Filmmaker Malcolm Le Grice observed, “It is perhaps in the continued development of the recurrent, more radically abstract theme of ‘chromatic music’ that the digital media could make their most significant contribution” (Le Grice 2001).
12.6 Conclusion
A history of the evolution of the relationship between the aural and the visual realms is one of scientific discoveries, evolution of technology, perception studies, and artistic results. The developments of technology in the twentieth century, and in particular the development of digital technology, have made finally explicit what has been called the audiovisual discourse. The panorama of artistic works that can be placed inside this discourse is not at all uniform in terms of form, content, or technology used. The writing of code, however, has been a key element in providing a unifying approach.
______________
Part of this chapter draws on E. A. Edmonds and S. Pauletto, “Audiovisual Discourse in Digital Art,” SIGGRAPH 2004 Electronic Art and Animation Catalogue (New York: ACM Press, 2004), 116–119. Exhibition catalog.
1. Port Hacking was in the film section of the Sonar 2004 Festival in Barcelona.
2. These were shown, for example, at the SIGGRAPH 2004 art exhibition, Los Angeles, and the GRAPHITE 2004 art show, Singapore.
References
Bann, S. 1972. Introduction to Systems, 5–14. London: Arts Council of Great Britain. Exhibition catalog.
Baudelaire, C. 1982. “Correspondences.” In Les Fleurs du Mal, translated by Richard Howard, 15. Boston: David Godine.
Brown, A. 2006. “Code Jamming.” M/C Journal 9 (6). http://
Drew, J., and M. Harrison. 1986. “Eye Music: The Graphic Art of New Musical Notation.” Exhibition booklet.
Edmonds, E. A. 2003. “Logics for Constructing Generative Art Systems.” Digital Creativity 14 (1).
Edmonds, E. A., and S. Pauletto. 2004. “Audiovisual Discourse in Digital Art.” In SIGGRAPH ’04 ACM SIGGRAPH 2004 Art Gallery, edited by Sue Golifer, 116–119. New York: ACM Press.
Gage, J. 1993. Colour and Culture: Practice and Meaning from Antiquity to Abstraction. London: Thames and Hudson.
Jewanski, J. 1999. “What Is the Colour of the Tone?” Leonardo 32 (3): 227–228.
Le Grice, M. 2001. Experimental Cinema in the Digital Age. Wiltshire, UK: Cromwell Press.
Peacock, K. 1988. “Instruments to Perform Colour-Music: Two Centuries of Technological Instrumentation.” Leonardo 21 (4): 397–406.
Peacock, K. 1991a. “Famous Early Colour Organs.” Experimental Musical Instruments 7 (2): 1–20.
Peacock, K. 1991b. “Famous Twentieth-Century Colour Instruments.” Experimental Musical Instruments 7 (3): 1–20.
van Campen, C. 1999. “Artistic and Psychological Experiments with Synesthesia.” Leonardo 32 (1): 9–14.
Zielinski, S. 1999. Audiovisions. Amsterdam: Amsterdam University Press.