Synesthetic Instrument

A machine for performing sound and image

c23-fig-0162.jpg

162. In the Web-based performance In C (2015) by Luisa Pereira, Yotam Mann, and Kevin Siwoff, a group of performers use mice and keyboards to control the pace at which their individual instruments advance through a graphical score, producing highly variable musical results.

Brief

Create an “audiovisual instrument” that allows a performer to produce tightly coupled sound and visuals. Your software should make possible the creation of both dynamic imagery and noise/sound/music, simultaneously, in real time.

You are challenged to create an open-ended system in which sonic and visual modalities are equally expressive. Its results should be inexhaustible, deeply variable, and contingent on the performer's choices, and the basic principles of its operation should be easy to deduce, yet also allow for sophisticated expression. Interactions with your instrument should generate predictable results.

Assume your instrument receives input from the actions and gestures of a performer. Will they use a keyboard, mouse, multi-touch trackpad, pose tracker, or a less common sensor? Select (or construct) your instrument's physical interface with care, giving consideration to its expressive affordances. Categorize the data streams it provides: are these continuous values, or changes in logical states? Do they have a perceptible duration, and are they persistent or instantaneous? Are they one-, two-, three-, or four-dimensional?

Assume your instrument generates output for a graphic display and audio system. Think through the possibilities offered by different visual variables like hue, saturation, texture, shape, and motion, and different auditory elements including pitch, dynamics, timbre, scales, and rhythm. Link these sonic and visual elements together by establishing mappings between your system's input and output. For example: the faster the performer moves her cursor, the brighter her cursor appears, and the higher the pitch of a synthesized tone. Such mappings help set up expectations that can be manipulated by a performer to create contrast, tension, surprise, and even humor.

Attune us to your instrument's unique expressive qualities by using it in a brief performance. One suggestion for structuring this performance is to begin with an expository demonstration of your instrument's interface, guiding the audience into understanding how it operates before presenting more complex material. The repetition and subsequent elaboration of themes is also a helpful compositional strategy.

Learning Objectives

Variations

Making It Meaningful

Good instruments offer inexhaustible possibilities for expression, composition, and collaboration. To a performer, the value of an instrument hinges on how well it supports the creative feedback loop known as “flow.” i An instrument that is responsive but crunchy can be more gratifying than a device that easily begets dazzling results. Hence, in this assignment, the emphasis is on the suppleness of an instrument's interaction design and the range of expression it makes possible (the performer's experience), rather than on the aesthetics of the audiovisuals it produces.

A “North Star” for instrument design is to create something “instantly knowable, yet infinitely masterable.” ii Consider the pencil, or the piano: its basic principles of operation are simple enough for a child to deduce, yet one can spend a lifetime using it and still find more to say; sophisticated expressions are possible, and mastery is elusive. From the standpoint of systems design, our challenge is that ease of learning and expressive range are antithetical design requirements: optimize for one, and the other suffers. In making an instrument for simultaneous sound and image, this challenge is compounded by another: expressive malleability in one modality often comes at the expense of rigidity in the other.

It is helpful to consider the typology of audiovisual systems and the design strategies they use. Sound visualization, for example, is a common feature in desktop music players, VJ software, and phonology tools. Principles of image sonification underpin film scores, game music, and some tools for the visually impaired. The term “visual music,” used by creators of some color organs and abstract films, can even refer to a strictly silent medium, one comprised solely of animated imagery with temporal structures that are analogous to musical ones. In the realm of computer-based performance systems, designers have used a variety of visual interface metaphors to control and represent sound. “Control panel” interfaces, for example, use knobs, sliders, buttons, and dials to govern synthesis parameters, evoking the look of vintage synthesizers. “Diagrammatic” interfaces, such as scores and timelines, use the graphical conventions of information visualization to organize representations of sound along axes like time and frequency. Others use “audiovisual objects,” wherein a performer stretches, manipulates, or knocks virtual objects together in order to trigger or modulate corresponding sounds. In “painterly interfaces,” gestural marks performed on a 2D surface conjure and influence a fine-grained aural material.

c23-fig-0163.jpg

163. Amit Pitaru's Sonic Wire Sculptor (2003) is a tablet-based system for authoring looping scores. Melodies are represented by drawings that curl around a cylindrical 3D space.

c23-fig-0164.jpg

164. Psychic Synth (2014) by Pia Van Gelder is a responsive audiovisual environment in which a participant's brainwaves are captured by an EEG headset, in order to establish an immersive biofeedback loop that governs video projection, colored light, and immersive sound.

c23-fig-0165.jpg

165. Patatap (2012) by Jono Brandel and Lullatone is a browser-based app for the simultaneous performance of sound and animated imagery. Each key on a standard computer keyboard triggers a unique noise and snappy animation.

c23-fig-0166.jpg

166. Jace Clayton's Sufi Plug Ins (2012) are a suite of seven free tools that extend the functionality of Ableton Live, a commercial music software sequencer. These software-as-art visual interfaces are designed to support non-Western conceptions of sound, such as North African maqam scales and quartertone tuning. The interface is written in the Berber language of Tamazight, using its neo-Tifinagh script.

c23-fig-0167.jpg

167. Orca (2018) by Hundred Rabbits is an esoteric programming language and live-coding interface for creating and performing procedural sound sequencers.

Additional Projects

Readings

  1. Adriano Abbado, “Perceptual Correspondences of Abstract Animation and Synthetic Sound,” Leonardo 21, no. 5 (1988): 3–5.
  2. Dieter Daniels et al., eds., See This Sound: Audiovisuology: A Reader (Cologne, Germany: Walther Koenig Books, 2015).
  3. Sylvie Duplaix et al., Sons et Lumieres: Une Histoire du Son dans L’art du XXe Siecle (Paris: Editions du Centre Pompidou, Catalogues Du M.N.A.M., 2004).
  4. Michael Faulkner (D-FUSE), vj audio-visual art + vj culture (London: Laurence King Publishing Ltd., 2006).
  5. Mick Grierson, “Audiovisual Composition” (DPhil thesis, University of Kent, 2005).
  6. Thomas L. Hankins and Robert J. Silverman, Instruments and the Imagination (Princeton, NJ: Princeton University Press, 1995).
  7. Roger Johnson, Scores: An Anthology of New Music (New York: Schirmer/Macmillan, 1981).
  8. Golan Levin, “Audiovisual Software Art: A Partial History,” in See This Sound: Audiovisuology: A Reader, ed. Dieter Daniels et al. (Cologne, Germany: Walther Koenig Books, 2015).
  9. Luisa Pereira, “Making Your Own Musical Instruments with P5.js, Arduino, and WebMIDI,” Medium.com, October 23, 2018.
  10. Martin Pichlmair and Fares Kayali, “Levels of Sound: On the Principles of Interactivity in Music Video Games,” in Proceedings of the 2007 DiGRA International Conference: Situated Play, vol. 4 (2007): 424–430.
  11. Don Ritter, “Interactive Video as a Way of Life,” Musicworks 56 (Fall 1993): 48–54.
  12. Maurice Tuchman and Judi Freeman, The Spiritual in Art: Abstract Painting, 1890–1985 (Los Angeles: Los Angeles County Museum of Art, 1986).
  13. John Whitney, Digital Harmony: On the Complementarity of Music and Visual Art (New York: Byte Books/McGraw-Hill, 1980).
  14. John Whitney, “Fifty Years of Composing Computer Music and Graphics: How Time's New Solid-State Tactability Has Challenged Audio Visual Perspectives,” Leonardo 24, no. 5 (1991): 597–599.

Notes