23 Rebecca Fiebrink Uses Movement to Generate Sound
Why not forget about programming and instead build new instruments from examples of movement and sound?
—Rebecca Fiebrink113
Rebecca Fiebrink’s mantra is “It’s faster to build a function from examples rather than write code.”114 By a function she means the mathematical methods to express how a computer produces sounds. This can be very complicated. There are many different pathways for sound input and output, represented in complex mathematics. But what if you have something in mind that doesn’t obviously operate in mathematical terms? What if you want to build an instrument you can control with your movements? “That’s an example of embodied creativity, and it’s very difficult to make a mathematical function for this,” Fiebrink explains.
Fiebrink’s challenge is to “build interfaces with different types of technologies which allow us to express ourselves in new ways or allow more people to participate in music making.”115
She discovered computers at the age of three when her father brought home a Tandy computer, one of the earliest PCs from the 1980s. But she was soon equally entranced with the flute and piano and “always assumed she would be a musician.”116 Then her high school offered a computer-programming class for the first time. She soon realized she was good at programming and could do some interesting creative tinkering.
She went on to study computer science at Ohio State University. But she wanted “a nontypical computing career,” which would include music.117 As it happened, there was a group in computer science interested in empirical digital musicology, studying the notation in scores to test hypotheses about how people compose and how music evolves over time. Fiebrink joined a graduate seminar taught by David Huron, who used machine learning to grade the difficulty of playing a piece of music.
Inspired, she invented what she calls the Wekinator, named after Waikato Environment for Knowledge Analysis (Weka), software developed in Waikato, New Zealand. The Wekinator is easily accessible software that does not require a knowledge of computer programming for people who wish to use off-the-shelf learning algorithms.
As she puts it, “Why not forget about programming and instead build new instruments from examples of movement and sound? So I say, ‘Hey, computer, when I do this …’”118
Anyone can use the Wekinator to create original music and performances. Each person who uses it starts by training it to follow their gestures. The machine reads from a webcam trained on the performer. Thus a closed fist might generate a specific pattern of drumming from a drum machine and an open palm a different sort of pattern.
The aim of conventional machine learning is to create a faithful model of data, such as a database of medical diagnoses to figure out what’s wrong with a patient and what drug would be effective. “Most machines,” says Fiebrink, “won’t let you change the data. Why would you? That’s sort of cheating. You are taught that you don’t change the data.”119
The Wekinator is very different. It’s open source. People download the software from the web and “very easily make new data on the spot and change data over time—for example, use it to build a new gesturally controlled instrument,” she tells me. “The ability to edit data and change the model I refer to as interactive machine learning.”120
For all of this, there is no need to know how to program. Fiebrink has received many positive responses applauding how easy the Wekinator is to use. Watching her set it up and straightaway produce results is inspiring and astonishing.121 Fiebrink describes the Wekinator as a “metainstrument,” because it allows users to create new instruments with no need for programming. Having mastered it, users no longer need to think about algorithms but can focus on “using them to achieve a creative vision.”122
Fiebrink has also invented Blotar, which has a wider sound palette and is capable of playing continuous sounds, rather than just staccato drum beats. As she moves in front of Blotar’s webcam, Blotar produces continuously changing sounds—modulating from a flute to an electric guitar, for example. Performers have developed very subtle hand movements to elicit extraordinary, otherworldly sounds from the machine.
Thus Fiebrink is working toward realizing her ambition of empowering people to create music—people with little or no musical education, as well as those who may be physically impaired. She hopes that the next generation of content-generation algorithms—capable of yielding a great amount of content for a small investment of data—will yield “surprises in musical composition.”123
Musicians already report that these instruments surprise them in their explorations, catalyzing their creativity.124 Many performers now use Wekinator and Blotar in their performances to create a mesmerizing amalgam of dance and sound in which one sparks the other.