35 Hannah Davis Turns Words into Music
It’s a different way of listening to music, a different way of thinking about it.
—Hannah Davis152
Music stirs the emotions. Sometimes just a chord will do it. Literature too works on an emotional level. Hannah Davis’s project is to bridge music and literature, to bring sound to our experience of reading a book.
Davis began programming as a child and studied music as well. For a few years, she studied international relations and spent some time working in Ghana. Then she renewed her interest in computer science and studied for a master’s degree in creative communication technology at the Interactive Telecommunications Program (ITP) at NYU.
She studied data sonification, transforming data—in this case, the grammar and structure of a novel—into sound. To do this, she fed texts by Ernest Hemingway, Virginia Woolf, and David Foster Wallace into a computer and had it tag every word with the appropriate part of speech. Her program picked up the rhythms of Hemingway’s distinctive writing style—short sentences with few descriptive words—and Woolf’s melodic poetry. “Hemingway was staccato—I love that—while Woolf’s poetry has a rhythm falling back on itself.” Her program also caught David Foster Wallace’s long, flowing sentences.153
But she was less interested in text analysis than in creating music that could reflect the moods of the novel. The computer lexicons available, however, were very limited. They contained too few emotion words, only a few hundred, and all obvious—happy, sad, and so on—and covering too few emotions, whereas authors are far more nuanced in their evocations of emotions.
Then Davis discovered Saif M. Mohammad’s EmoLex, the NRC Emotion Lexicon.154 This lexicon contains fourteen thousand commonly used words tagged according to the emotions they evoke by thousands of people recruited via Amazon’s Mechanical Turk. The eight emotions it identifies are anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. Using this lexicon, Davis worked with Mohammad to create a program she calls TransProse, which trains a computer to detect emotion in text. TransProse also divides the emotions in the text into positive and negative. Thus EmoLex associates the word abandonment with anger, fear, and sadness, all negative sentiments.
This textual data forms the emotional profile of the novel. Davis then transfers the data to a music-processing library equipped with the necessary music theory—tempo, scale, octave, and notes. A quick tempo and major keys are expressive of happiness, whereas minor keys communicate sadness. The system is equipped with sound fonts so that it sounds like a piano. Then it is converted into wave forms for MP3 encoding.
The scores follow the trajectories of the novels, reflecting the density of the eight emotions and two states—positive and negative—throughout the text. Conrad’s Heart of Darkness, full of a sense of approaching doom, produces a somber, dark score, whereas the score for Barrie’s Peter Pan, with an emotional profile of trust and joy, is light and cheerful.155 Thus TransProse generates “piano pieces whose notes are dependent on the emotion words in the texts.”156
In 2016, Davis was commissioned to create a score for a fifty-piece orchestra, to be performed inside the Louvre Pyramid in Paris. The sponsor was Accenture Technology, a branch of Accenture, a business and technology consulting firm that wanted to show the breadth and potential of AI and its transcendence across cultures. What better way to do this than with music? A member of Accenture saw Davis give a lecture and realized she was exactly what they were looking for.157 “Words have real meaning, and they can make music,” as Mark Knichrehm, group chief executive at Accenture Strategy, another branch of Accenture, put it.158 Davis provided the perfect alchemical mix of human and machine to generate the music for the event.
Davis used her TransProse system to analyze the emotions communicated by thousands of news articles, covering business topics such as cybersecurity and the internet of things. She digitized the data for each of the eight emotions and divided it into separate files for particular instruments to create a series of musical motifs. In collaboration with composer Mathieu Lamboley, she created Symphonologie: The Music of Business, with three movements: “Voices of Business,” “The Rise of Technology,” and “A New Digital World.” The symphony was performed on September 20, 2016, and was a dazzling multimedia experience, with the music accompanied by a spectacular data visualization generated in real time.159
This is clearly the sort of approach that is perfect for film scores. Davis’s present research includes looking into automatic film scoring to generate film scores directly from visual material.160 For this project, she uses neural nets for face/character and scene detection. If she succeeds, she may well change the future of composing scores for films.
There has always been a “problem with the way the media talks about my stuff,” says Davis. One newspaper wrote, “‘Robot creates music based on a novel.’ This makes people a little bit more defensive than they need be. … Dystopian scenarios are really not interesting.”161
She looks forward to a future enhanced by AI, with a basic income for all and a good transition plan for how to move from a world dominated by us to one where machines do much of the work. Davis feels that it would be an excellent thing for humanity if robots took over our jobs, allowing us to redefine ourselves in other ways. She adds that she definitely believes that computers can be creative.162