8 Sound and Meaning

Every single sound, no matter how subtle or simple . . . is considered, composed [and] orchestrated to have an effect on the audience.

—Mark Mangini, sound designer (quoted in Horn 2016)

What and how do sounds mean? We’ve now got a grasp on what sound is, how to use sound libraries, and how to record, edit, and mix sounds; but how do we know what sounds to use, and where? Understanding how sound creates meaning will help you to understand how best to use sound to creative advantage in your projects. A very creative use of sound can also sometimes make up for less technical proficiency.

There are many theories about sound and its meanings that come from different disciplinary perspectives (psychology, philosophy, musicology, and linguistics, for instance). It’s not my intent to cover all of the theories of sound and meaning, but understanding some of the major theories about sound and meaning will help you to understand, talk about, and use sound more effectively and more creatively. At the heart of each of these theories is the concept that sounds are complex and evocative: that is, they are never just “sounds,” but become associated with other things through our daily experiences of them, and these associations are critical to our understanding of how to use sound as a component of design.

8.1 Conditioning

Conditioning is a psychological behavioral process in which our response becomes learned over time as a result of being reinforced by repetition or extreme circumstances. The classic example is the Russian physiologist Ivan Pavlov, who in the 1890s ran studies of his dogs responding to being fed. In these studies a bell was rung every time the dogs were fed, and the dogs came to associate the sound of the bell with food. Pavlov could then ring a bell and the dogs would salivate. Today, we continue to use conditioning in training dogs for bomb and drug sniffing, but humans are also trained to respond to certain events in certain ways, consciously or not.

Conditioning is used in product, film, and game sound to make the listener associate sound with certain events: watch the movie Jaws (1975), and notice how we’re conditioned early on that the famous musical cue represents the shark. When we become conditioned to expect the association, that link is broken on purpose, and the shark comes without warning, leaving us off-balance and adding tension. In video games, we’re often taught to associate certain sounds with certain events—that was a bad poison we just drank, or it’s good to collect coins. The association of sounds with events helps teach us or orientate us in the game, reducing the learning curve. And we’re conditioned in other aspects of our lives by sounds—have you ever been out somewhere and heard someone else’s phone with your ringtone? Did you jump or grab your own phone?

With a number of colleagues at the University of Waterloo I’ve studied the use of sound in slot machines to manipulate and control players. In slot machines, of course, we only ever hear winning sounds: we don’t hear losing sounds. But we discovered that slot machines also play winning sounds when we lose sometimes, what we called “losses disguised as wins” (see Dixon et al. 2014). A loss disguised as a win is when, for example, we put in a $2 bet and win back a dollar. Technically, we lost a dollar, but the sound tells us we won and plays the same winning sounds as a real win. We respond, physiologically, to these losses as if they were wins, because the winning sound conditions us to hear them as wins. In this way, sounds are deliberately manipulative as they play on that conditioning effect.

It doesn’t need to be exactly the same sound each time to evoke a conditioned response. Similar sounds can affect our perception of a sound: one scientific study conditioned mice to associate closely related sounds with trauma (see Aizenberg and Geffen 2013). By playing a similar sound, they could invoke a similar response in mice.

The theory of classical conditioning—the Pavlovian notion that we could be conditioned to respond in a particular way—gave way to a more complex understanding of this process under B. F. Skinner, who introduced the idea of operant conditioning. With operant conditioning, there are positive and negative reinforcements, and positive and negative punishments. “Positive” and “negative” refer to the presence or absence of something, rather than good or bad. For instance, a positive reinforcement is a reward—like a sonic reward in a game. A negative reinforcement is the removal of something—like the seatbelt sound your car makes until you do the desired behavior (plug your seatbelt in). When it comes to punishment, a positive punishment is adding a negative consequence to decrease a behavior—like an angry buzzer sound in a game show when you do something wrong. A negative punishment is to take away something good to reduce a behavior, like revoking your access to music that you love until you change your behavior.

Exercise 8.1 Conditioned to Sound

How many sounds can you list that you are conditioned to respond to in a particular way? What is your association and why? What types of conditioning are they (positive/negative, reinforcement/punishment)?

Exercise 8.2 Conditioning Sounds

Try to imagine some creative uses of the conditioning effect beyond some we’ve already talked about (hint: think about products, advertising, etc.).

8.2 Sonic Archetypes, Stereotypes, and Generalizations

Archetypes are a kind of ideal example of a concept, and show up in dreams, literature, religion, and art across the world: psychoanalyst Carl Jung, with whom archetypes are often associated, called them primordial images. For Jung, these archetypes show up as personality types or patterns of behavior, like the hero, or the wise old sage. These differentiations between types of people are often used in writing for fiction. They each have a defined main set of desires, fears, and talents. Because they are so often used in storytelling, it’s useful to understand them and think about how we might represent these people in sound.

Here I use archetype in the broader sense of the term, as patterns or prototype examples that can represent a phenomenon. Some of our archetypes have developed through evolution and biology, and others are cultural. Some sounds carry universal meaning to all humans everywhere (global would perhaps be a better term). Low-frequency sounds are typically associated with threat, because those are the sounds that signify danger in nature: thunder, volcanoes, earthquakes, large mammals. High-frequency sounds are not usually associated with threat: small birds and the like. There are exceptions, of course. When high-pitched sounds are associated with threat, they are typically of the psychological disturbance variety, such as Bernard Herrmann’s strings in Psycho (1960) (see Collins and Tagg 2001). These sounds have become archetypal in film: it’s very rare to have a scary scene without a low-frequency hum or a screeching high frequency accompanying it.

We also have conventional sounds: sounds whose meaning we agree on through cultural convention, not through some universality. For instance, the sound of a phone ringing: there is nothing inherent in the sound to indicate that a phone is ringing, but everyone in the West understands what a ringing phone sounds like (a doorbell ringing is another example). Our conventions tell us what the sound means—what it refers to, or what it connotes. Conventional sounds can be archetypes as well, although their meaning is specific to specific audiences. The sine waves accompanying early computers as warning beeps became standard interface sounds in science fiction, and remain so today, even though we could use any sound now.

Beyond these universal and conventional sounds that are biologically or culturally reinforced, we also have sounds that are personal to us based on our past associations with them, as we saw with conditioning. On a personal level we associate sounds with certain events in our own lives. We can use the term anamnesis to describe this phenomenon, the “involuntary revival of memory caused by listening and the evocative power of sounds” (Augoyard and Torgue 2009, 21). These may be shared by groups of people who experience similar events, but may not be a widespread cultural phenomenon. There are times I’ve heard a sound similar to the sound Super Mario makes when he jumps and gets a coin, and I immediately am reminded of playing the game with my younger brother.

If we are designing sound, it’s important to know our main audience. Composers and sound designers often use sonic symbols that signify other cultures, but those cultures can be offended by such generalizations or have different associations with those sounds: perhaps every time we go to “Australia” in a sonic narrative, we hear the didgeridoo, for instance, but Australians may associate the didgeridoo not with Australia, but rather with the indigenous Northern Australian population specifically. It’s important, then, to understand who our audience is and what associations they may have with the sounds. We can’t control the anamnesis, or personal responses to sounds, but we can control which sounds we choose, based on how people will respond to or understand conventional or universal sounds.

Exercise 8.3 Universal and Conventional Sounds

Listen to different aspects of a soundscape you’ve recorded. What aspects are conventional and which are universal? Download thirty random sound effects and categorize them as universal or conventional. Explain your category choice. Are there any features of the sounds that are unique to conventional or universal sounds?

Exercise 8.4 Sonic Generalizatons and Stereotypes

What sonic generalizations can you recall that represent places, cultures, religions, seasons, periods of time, or other aspects in film, television, radio, or games? Think beyond just culture: for instance, every time we see a bicycle on screen in a rural setting we tend to hear a bicycle bell. Nighttime often has the sound of crickets, or if it’s urban, a police siren. What sounds are associated with women? What sounds with men? How do the sounds of women differ from those of men, and why?

Exercise 8.5 Anamnesis

Are there sounds that you know are not conventional or universal, but which affect you on a personal level by evoking some past memory? I associate a mourning dove with my grandmother’s house, for instance, since when I stayed at her house I would often hear them. Make a list of all the sounds you can think of that cause anamnesis for you, along with their associations. Keep adding to the list as you think of more over the coming days and weeks.

Exercise 8.6 People Sounds

Sit in a public place and watch people in the vicinity, and mentally assign them their own sound effects. Alternatively, choose people from your own life and think of them in terms of their own sound effects—imagine assigning them a sound effect as a ringtone identifier. What did you choose and why?

8.3 Basic Semiotic Theory

Semiotics (sometimes called semiology) is the study of signs and symbols, and has developed from linguistic theory as an understanding of communication and language. It can get quite complicated, but it’s a useful way to understand sound as a form of symbolic, rhetorical, communicative language. The field uses different terminology depending on which theorist we’re looking at, which can make semiotics confusing. We’re going to stick to a very simple explanation here, but semiotics can be a really fruitful theory to use to talk about and analyze the use of sounds in media, and if you’re interested in how sound communicates meaning on a theoretical level, I’d recommend you dig a little deeper into this subject.

In the semiotics of Swiss linguist Ferdinand de Saussure, anything that represents something else can be referred to as a sign or signifier. What that sign refers to—the idea, concept, or meaning—is known as the signified. If we take the sign of the sound of a phone ringing, the ringing sound is the signifier, and the signified is that someone is calling. We could take this further: if, for instance, I am waiting for a dreaded call from a doctor to find out if that lump is malignant or not, the signified might also include anxiety or other emotions. If we imagine a film scene in a thriller where something bad is about to happen and a car alarm goes off, the car alarm might signify that someone is breaking into the car (or just bumped it), but it can also be a signifier of alarm and distress for the character in the scene about to get attacked.

The terms denotation and connotation are used to describe the relationship between signifier and signified. In the above example with a phone ringing, the denotation was “someone is calling,” and the connotations were anxiety and dread. The denotation is the literal meaning of the signifier, and connotations are the secondary associations we have with the sign. In everyday use, we don’t often speak of denotation, because that is usually quite obvious, but the connotations are what are important in sound design. What a sound means—beyond its literal representation—can add so much to a scene. Imagine a film scene in which a character is getting increasingly angry while having an argument in their kitchen. The simultaneous use of a kettle boiling and coming to a scream might connote the feelings of the character.

The fact that sounds may carry information means that they may also cause a communication failure of interpretation. We may not have the same associations and connotations as our audience, so it’s important to understand that whatever signs we use, the audience understands the meaning. As Umberto Eco puts the matter: “To make his text communicative, the author has to assume that the ensemble of codes he relies upon is the same as that shared by his possible reader. The author has thus to foresee a model of the possible reader . . . supposedly able to deal interpretatively with the expressions in the same way as the author deals generatively with them” (1979, 7). Putting that into our own terms, the listener first has to understand that we’re using sound as a metaphor or symbol, and second has to understand the sound with the same interpretation as we would (on a conscious or subconscious level). For instance, they would have to understand not only what a kettle boiling sounds like, but also the association of boiling over with stress or anger.

As well as sharing the same symbols, the listener and sound designer must share the same expectations and sociocultural norms. In other words, the listener and sound designer must agree that the symbols represent the same sociocultural ideas, and the listener must have an adequate response to those ideas. For example, if my father heard the heavy thrum of a rap song’s beat, he might associate this concept with the teenage Jeep jockeys with massive woofers who drive by his house late on a Friday night and wake him up. Although he therefore may have the understanding to hear the sound as representative of some form of power, my father will likely hear this as somebody imposing their power on others, while the teens may hear this as making them feel powerful.

Ultimately, when we use sound, we should think about its potential meanings for different audiences. If we have a personal association with a sound, we can’t assume everyone has that same association; we have to consider other potential meanings. Most of the time, sound designers do this intuitively, but consciously thinking about sound’s potential meanings or connotations can be a fruitful way to explore the use of sound as a metaphor.

Exercise 8.7 A Sonic Warning

A classic visual design exercise is to draw a sign that anyone in the far future could understand to mean that a place is very dangerous, like a nuclear waste disposal site. The design problem is what images one could use to signify the danger of the site in order to be universally understood. How would you use sound to represent a universal warning for the future? Make that sound. What effects did you rely on and why?

8.3.1 Semiotic Analysis and Reference Material

While we may not consciously use semiotics in our sound design practice, we may find ourselves using some of its techniques without realizing it, particularly when trying to think through why a particular sound or particular group of sounds evokes in us a particular response. We can analyze what sounds mean by comparing uses of that sound to similar scenes or similar sounds in other media. Musicologist Philip Tagg (2003) calls this interobjective comparison (that is, comparing between objects, or texts, like books, film, and radio). For instance, if we find the same doomy bass sound right before a murder in six different movies, we could probably surmise that the doomy bass signifies murder, threat, and anxiety. We might also gather similar sounds from sound libraries and compare the terminology used to describe the sound: if the sound effects were given names like “murderous bass” or “threat,” these are in the same general connotative sphere, so we could gather multiple examples of how people associate certain sounds with certain responses. If we want to signify an alien landscape, the first place we might look are sound libraries with tags like “alien.” We might go watch and listen to sci-fi radio plays, movies, or television. That way, we have an idea of how other people have already created a template of signification for us to follow.

Another approach involves a comparison not between objects or uses of sounds but across audience responses. These are intersubjective comparisons, and take a bit more work, because they require tests on a real audience. Years ago, I explored sound connotations by using reception tests. These are kind of like an auditory Rorschach inkblot test: you play a sound, and then ask people to respond freely to what the sound connotes for them. In my case, I played them common sounds used in science-fiction cinema, and asked them to give the sound a name, describe what made the sound, and any associations they had with the sound (Collins 2002). Many responses described the physical causal relationship they heard in the sound: “stampede on sheet metal,” “metal and ironmongers,” “arrow shot from laser gun.” Such images indicate that, far from sounding physically detached, sound samples are “seen” in the mind as having activity and motion (we’ll come back to that below). In fact, remarkable in my study that a single sound would call up in subjects’ imagination a specific film. This told me that single sounds can carry a lot of weight when it comes to signification. The listeners weren’t aware of where the sounds came from or what I was looking for, but many found connotations very close to the meanings I had perceived myself, and confirmed with the interobjective methods I’d used as well.

Trevor Cox, a researcher at Salford University in the UK, ran a study online a number of years ago called BadVibes that asked people to vote on the most horrible sounds they could imagine. By collecting a lot of data about how people respond to certain sounds, Cox could determine what sounds people associated with particularly unpleasant feelings (the sound of vomiting came out ahead overall). Cox is often running more experiments on his website, sound101.org, and it’s always worth spending a few minutes to help out researchers, because that information eventually helps feed back into training new practitioners. It also helps you to see how our understanding of sound as a communicative tool is developing.

Exercise 8.8 A Semiotic Study of Sounds

Find a sound composition: it could be a radio play, or the sound to a movie (without using the images), or a musique concrète composition. What are the important signifiers in the piece, and what do you think they signify? How do they relate to other signifiers used? Can you find similar sounds used somewhere else that support your idea of what the sounds connote (interobjective comparison material)? What stereotypes do you hear, and which are cultural or universal? Carry out a reception test among your friends, colleagues, or others and have them freely respond to the sounds. Did they all have the same connotations? What were the differences, and why do you think those differences existed? What happens to the sound’s meaning when you put different effects on the sound?

Exercise 8.9 A Semiotic Study of a Soundscape

Create a soundscape, for example, the basement of a serial killer’s home. Repeat your semiotic study as described above. What responses did you get, and how did you use sound to help evoke those responses? What effects did you add to any recorded sounds, and what impact did those effects have?

Exercise 8.10 What Do I Mean?

Create an alien sound language and devise a short message. See if anyone can interpret it. What research reference material did you use and why? How did you test its meanings?

Exercise 8.11 Musique Concrète

Musique concrète was a style of music that used recorded sound effects as raw material, back in the days of tape, and is exemplified by the works of Pierre Schaeffer and Pierre Henri. Listen to a piece of musique concrète and freely associate to it by writing down any imagery that pops into your mind as you listen. Listen to the piece again—what sounds or effects led to that imagery? Create your own piece of musique concrète that relies on sound objects that you think have specific connotations, then try it out on an audience and ask them to freely associate to the piece. Were you able to evoke the imagery you wanted? Why or why not?

8.4 Phenomenology, Embodied Cognition, and Intersensory Integration

One other major group of theories that is useful for our thinking about sound and its meanings relates to areas of philosophy and psychology that have tried to understand the senses and the body in relation to the world and its meanings: phenomenology, embodied cognition, and intersensory integration. These are all related, as they all deal with our senses and our understanding of the world through those senses.

Phenomenology is a branch of philosophy relating to how human experience is mediated by our sensory perceptions and our bodies. Phenomenologists hold that our mind and how we think about the world is shaped by our physical experience and existence in that world. German philosopher Martin Heidegger (1962), for example, believed that our experiences are predicated on our being embedded in our environment (what he calls being-in-the-world). In this way, how we think is shaped by our interactions with the world. French phenomenologist Maurice Merleau-Ponty (1998) similarly held that all perception is understood through the ways in which we are able to act in the world and move around in our environment. In this way, the sensory inputs that we receive from the environment mold our understanding of the world. Our body acts as a mediator between the world and our consciousness.

Phenomenology has been used in music and sound studies to explore the ability of sounds to evoke imagery in our minds, as a means of explaining some of the semiotic connotations discussed above. For example, in exploring radio drama, Clive Cazeaux argues with the German film critic Rudolf Arnheim, who stated in the 1930s that radio “seems much more sensorily defective and incomplete than the other arts—because it excludes the most important sense, that of sight” (Cazeaux 2005, 158). Cazeaux used the phenomenology of Merleau-Ponty to argue that our five senses are not five discrete channels that gather data for the mind but, rather, that the senses operate in unity: we listen and simultaneously see and feel the drama. And although we always listen causally, we are simultaneously interpreting and associating sounds with other events, objects, and emotions from our experiences.

Some phenomenological approaches to music and sound suggest that sound alone can evoke bodily sensations of touch and movement: these views include the “mimetic hypothesis” (Cox 2001), “bodily hearing” (Mead 2003), and “vicarious performance” (Cone 1968), among others. If I asked you to perform out loud the action and sound from the infamous Psycho shower scene, you’d probably lift your arm and say “eee—eee—eee.” I’ve seen people use that motion and sound as shorthand for saying someone is a little unstable. But if you watch the scene, you’ll see that actually the stabbing is not timed at all with the screeching strings. So why do we remember that sound as being linked with that action? The answer has to do with kinesthetic sympathy: even if we don’t play the violin, we have an understanding of how the violin is played, and we hear the action of the players in the music—a very stabby action! When we remember that scene, with the action of seeing the stab, even though it’s not timed to the strings, we remember them as being tied because of our bodily hearing of the music cue.

We understand human-made sounds (including those of playing a musical instrument) in terms of our own experience of making similar sounds and movements. That mental re-creation of the sound causes a neuronal and sensorimotor response that mimics the performer’s actions, and then we are able to interpret the emotional inflections through a mental re-creation of that action. People therefore can give meaning to sound in terms of mentally emulated actions. Put differently, we mentally (and sometimes even physically) imitate the expressiveness of an action or feeling behind a sound, based on our prior embodied experiences of sound making. I mentioned this above in my own studies of sound effects—some people freely associated actions behind sounds in their descriptions of imagery evoked by sounds.

Embodied cognition is the cognitive science corollary to phenomenology, and is based on recent cognitive science being done with fMRI machines. Scientists are studying how our visual, auditory, and physical systems in our brains are all closely linked, and the result is that we understand sound through our bodies as much as our minds. In the past fifteen years or so, scientists have discovered that when we’ve experienced something, that experience fires off neurons in our brain, so if we encounter a grizzly bear, a bunch of neurons are going to fire in our brain. If, later on, we just see a movie of a bear, or hear that roar, almost all the same neurons fire. In other words, when we get a similar stimulus, our brain pretty much responds as if it’s happening to us, for real, again. This works so well that if we see something happening to somebody else, we still experience it in our brain to some extent as if it’s happening to us. So if we see someone else getting pricked by a needle, many of our neurons fire as if we are being pricked by a needle, which is why they’re called mirror neurons—we mirror the experiences of others (see Niedenthal 2007).

Phenomenology and embodied cognition provide support for theories of intersensory integration—the theory that our senses are not separate unique inputs, but have some overlap, as Cazeaux argued above about radio. Some people have quite literal sensory integration in that they have a neurological abnormality called synesthesia, in which they may “see” a sound or “hear” a color, for instance. Chromesthesia is a sound-to-color synesthesia in which heard sounds evoke an experience of color. While there are many theories as to why this unusual sensory experience occurs, we probably all experience the phenomenon to some degree. That is, we all have some intersensory integration—our senses are not discrete individual “inputs” to our brain, but are overlapped and integrated in our minds. Recent research has shown, for instance, that when your eyes move, your eardrums move too, bringing together sound and vision (Gruters et al. 2018).

Our senses are constantly exposed to stimuli from multiple sensory modalities (visual, auditory, vestibular, olfactory, taste, touch), and we are able to integrate and process this information to acquire knowledge of objects in our environment. The senses interact with one another and alter each other’s processing and ultimately our overall perception. We’re not going to focus on the actual interaction of sound and image in this book, but it’s important to understand that image changes the perception of sound (and vice versa); so we can’t just take an audio file we’ve created for a radio play and put it to images and expect the same responses from an audience.

The idea of intersensory integration with sounds means that the way our brains process sound relates to how we process our other senses at the same time in association with sounds, or because of our own bodily response to sound. For example, if you imagine lemons making sound, would lemons make a low-frequency, mid-frequency, or high-frequency sound? Most people would say that lemons make a high-frequency sound. But lemons make no sound, so how can we agree on what sound they might make? Now imagine yourself making that sound: close your eyes and sonify lemons. Our shoulders go up, eyes get pinched, lips draw back—we feel the sound in our body—the piercing high frequency is similar to what our body does when we taste very sour food like lemons. This demonstrates how sound and taste can be integrated in our minds, and those types of interactions occur between other senses as well. Marketing scientists have studied the association between sound and taste for years—“gotta have that crunch,” for instance: the sound of potato chips affects our sense of their taste (see Spence and Shanker 2010).

Researchers have done a lot of work on the interaction between sound and image, particularly from the fields of psychology and marketing, and also from film studies. For instance, the ventriloquist effect refers to the idea that we localize sound with an image, particularly if that image is moving, as when we see a ventriloquist dummy “speak” even if the voice is coming from elsewhere (see Choe et al. 1975). Another well-known example is the McGurk effect, in which visuals can alter our perception of speech sound. The McGurk effect occurs when the audio component of one sound is paired with the visual component of another sound, leading to the perception of a third sound (McGurk and MacDonald 1976). Perhaps most well-known is the Bouba-Kiki effect, a phenomenon of phonosemantics, in which certain visual shapes are associated with certain consonant and vowel sounds: “bouba” is generally drawn as rounded, and “kiki” as having hard angles. These associations have been demonstrated across cultures (Köhler 1947). An interesting recent study has demonstrated that the Bouba-Kiki effect also occurs when the visual domain is occluded and the participant is given shapes to touch, rather than see—in other words, sound-shape associations hold for the haptic-auditory associations as well as visual (Fryer et al. 2014).

While visuals most often influence our perception of audio, audio also affects our visual perception. For example, beeping sounds can create the illusion of visual flashes (Shams, Kamitani, and Shimojo 2000). In another test, known as the motion-bounce illusion, two identical visual targets will be perceived as crossing through each other in the absence of sound cues, but when a brief sound is added at the moment that the targets interact, a bias perception toward the targets bouncing off each other occurs (Sekuler, Sekuler, and Lau 1997). Hulusić et al. (2010) showed that sound effects allowed slow animations to be perceived as smoother than fast animations, and that the addition of footstep sound effects to visual-based walking animations increased the perception of animation smoothness in a virtual environment. Such use of cross-modal illusions are useful when we need to drop frame rates or reduce visual fidelity.

In addition to evoking images, sound can also evoke touch, or haptic perception. Auditory cues frequently occur when we touch or interact with objects, and these sounds often convey potentially useful information regarding the nature of the objects with which we are interacting (Gaver 1993). Research has shown that when these auditory cues are presented without the touch, they can provide sufficient information for people to assess the size of objects and even what material they are made of (Freed 1990; Wildes and Richards 1988). In fact, research has demonstrated the ability of auditory feedback to effectively convey information regarding various attributes of objects including material, shape, size, and the like (Gaver 1993). Some studies have shown that we are able to discriminate touch-produced sounds from different surfaces when the sound is presented in isolation of other feedback (Lederman 1979). The majority of research to date indicates that people can discriminate between different objects and surfaces on the basis of their sounds (Lederman 1979).

Since sound is a physical vibration, we can feel some frequencies as much as hear them: most commonly the low-frequency sounds have a very physical feel, and if we put our hands in front of a loudspeaker and crank up the bass, we can feel the air move.

The implications of embodied cognition and intersensory integration are shown that sounds are capable of evoking responses in our bodies and minds based on our past experiences, and as sound designers we can draw on that knowledge to guide a person toward a particular feeling or association. Nick Wiswell, sound designer for many racing games, explains:

If you’re racing a car in real life, an awful lot of the driver’s feel, and I’ve spoken to a lot of the race drivers about this, is not from the audio or it’s not from the steering wheel, in some cases. It’s the g-forces. You actually, you can feel the vibration of the wheels through the steering wheel. And you can feel how the car is reacting to the g-forces basically from the seat of your pants. You sort of get that natural vibration and sense of movement, all of which is completely lost in a racing game. We can use force-feedback steering wheels to simulate the tires, but a lot of the cues a driver would use to let them know that things are going to happen they don’t get. Audio is a big part of simulating that. When do you need to shift? Well, you can use audio. If you listen to the car’s engine, you can pick out when I hear this tone, which it reaches this pitch, that’s my time to shift. If you’re listening to the tire audio, we’ve got lots of cues in there that are like, “You hear this sound? You’re about to lock your brakes. You hear this sound? You’re about to start losing traction on the front tires.” This is if you’re in a rear-wheel drive car, it’s this sound and it’s in the rear tires.

There’s a lot of really interesting audio cues that we’ve put in there that are haptics-type feedback, but it’s an audio feedback cue that’s telling the player you’re about to do this thing, stop doing that. Or, it’s time to do something, so there’s a lot of information in there that gives you that sense, and we do have a lot of low-end content in our recordings that if you’ve got a good subwoofer can simulate some of that feel. But people say, “Ah, a race car went by me in a race, and it really hit me in the chest.” It’s like, “Yes, it’s 135 decibels. Of course it hit you in the chest.” I don’t get that when I’m playing it at home on my twenty-seven inch TV with little half-inch speakers. It’s like, no, the laws of physics say that could never happen. So how can we make it sound that impactful, or that big, without it actually being that loud. I would like to think that if you did actually crank out a game to be that loud, it would hit you in the chest, but it’s a physical response, and the subwoofer can play a part in that, and we can simulate a sort of pushing of the air. But unless it’s at those volume levels it’s never going to hit you in the chest like it does in real life. But there’s a lot of work we can do there on simulating loudness. . . . Over the years we’ve used various distortion and saturation effects to try and make the car feel as loud as it is, even when you’re not playing it at loud volumes. YouTube videos are mostly distorted, of cars, and people think that’s how they sound, so you’ve almost got to slightly push things in that direction to make people think that, “Oh, it’s so loud it’s breaking up.” We don’t want to distort it and clip it and make it sound bad, but there are things we can imply to really sell that sense of volume and loudness. (quoted in Collins 2016, 312)

8.5 Summary

Whether we talk about semiotics, embodied cognition, phenomenology, conditioning, or other approaches to understanding sound, they all have at their heart that sound doesn’t exist in isolation—it has associations beyond itself that we as sound designers can rely on. Analyzing our own responses to sound can be a powerful way to understand how we can use sound to get a desired response from our audience. While most sound designers do this intuitively, having a language to conceptualize and discuss this process with our design team can be a really fruitful way to discuss your work and why a particular sound works in the context of your product.

Exercise 8.12 Haptic Audio

Sit in a small space with a subwoofer and EQ your music to just play a series of low frequencies (<200Hz): can you feel the sound as well as hear it? Where in your body do you feel it? At what frequency does the sound become less physical and more cerebral?

Exercise 8.13 Sound as Shape and Color

What sounds match the shapes in figure 8.1 (adapted from Schafer 1992)? What colors should the shapes be? What does a harsh sound look like, compared to a soft sound? What is a cold sound? What is a warm sound? What is a yellow sound? A blue sound? What is a sour sound? A sharp sound? A round sound? Create or record the sounds and use any effects necessary to emphasize the correlation.

Exercise 8.14 Categorizing Sounds in Themes

Download thirty random sounds from freesound.org or from a sound library. Categorize them together according to broader themes (powerful/weak, bright/dark, good/evil, death/life, etc.): you can think about sounds according to colors, to shape (envelope), or connotation, for instance.

Exercise 8.15 Changing Meaning

Take a sound and use effects to change its meaning. Make it dirty, wet, sharp, muddy, sad, evil, spikey, round, etc. Take one sound from the previous exercise, and using effects, transform it into its opposite.

Exercise 8.16 Classroom/Partner Game Sound Charades

Now that we have developed a language for talking about sound, we can play sound charades: Take a sound from your list of sounds and describe it in any way other than using a word related to what causes it. So a bird chirping could be described as a bright, cheerful sound, or as a sound that signifies dawn or spring, or in terms of its acoustic properties, like a high-frequency sine wave sound in short staccato bursts of about 60 dB. Have other students try to guess what sound you’re describing.

Reading and Listening Guide

Karen Collins and Philip Tagg, “The Sonic Aesthetics of the Industrial: Re-Constructing Yesterday’s Soundscape for Today’s Alienation and Tomorrow’s Dystopia” (2001)

In this article, Philip Tagg and I explore the semiotics of sounds in industrial music, a genre that relies on a lot of sonic conventions of science fiction cinema. One of the things we explore is which sounds are associated with dystopia and which are associated with utopia in soundscapes of science fiction.

Marc Leman, Embodied Music Cognition and Mediation Technology (2007)

Leman’s book explores an embodied cognition approach to music research, bringing together research in cognitive science with research in computer science and musicology. While the book is focused on music and several chapters relate specifically to music technologies, the first half of the book provides an accessible overview of embodied cognition more generally. Leman outlines some of the issues with the more cognitivist approach to music that has dominated music study for centuries.

Trevor Cox, “Scraping Sounds and Disgusting Noises” (2009)

This article outlines Cox’s psychoacoustic internet experiment with horrible sounds, and ties the sound to our biological and anthropological history. It’s a fascinating read into why some sounds, like scraping nails on a chalkboard, evoke visceral responses from us. Cox also explores how visual stimuli might affect our response to those sounds, in “The Effect of Visual Stimuli on the Horribleness of Awful Sounds” in the same journal. What are your most and least liked sounds, and why? Undertake any experiments Cox has going on at his website: why do you think he’s testing those sounds, and what is the purpose of the experiments?

Philip Tagg, Music’s Meanings: A Modern Musicology For Non-Musos (2013)

Tagg’s thick opus is an overview of semiotics and musicology for non-musically trained people (non-musos). Although there isn’t a lot of work on sound effects, there is a thorough overview of Tagg’s approach to music semiotics. He provides whole chapters on intersubjectivity and inter-objectivity, on analysis. Regularly updated in the e-book version.

Don Ihde, Listening and Voice: A Phenomenology of Sound (1976)

Ihde is one of the leading figures in phenomenology, and though his writing in the 1970s is perhaps a bit too academic for today’s tastes, his work is an important contribution to understanding sound from a phenomenological perspective. In this book he grapples with the dominance of vision in our language and our approaches to phenomenology up to that point in time.