The term of medium (plural: media) covers a wide variety of phenomena: (a) TV, radio, and the internet (especially the WWW) as the media of mass communication; (b) music, painting, film, the theater and literature as the media of art; (c) language, the image and sound as the media of expression (and by implication as the media of artistic expression); (d) writing and orality as the media of language; (e) handwriting, printing, the book, and the computer as the media of writing. The definition provided by Webster’s dictionary puts relative order in this diversity by proposing two distinct definitions: (1) Medium as a channel or system of communication, information, or entertainment; (2) Medium as a material or technical means of expression (including artistic expression).
The first definition regards media as conduits for the transmission of information, while the second describes them as “languages” that shape this information (Meyrowitz 1993). (The use of quotation marks in this entry will distinguish “language” as a collection of expressive devices from language as the semiotic code that forms the object of linguistics.) The relevance of the concept of medium for narratology is much more evident for type 2 than for type 1. Ong (1982) has objected to a conception of media that reduces them to “pipelines for the transfer of a material called information.” If indeed conduit-type media were nothing more than hollow pipes for the transmission of artifacts realized in a medium of type 2 (e.g. a film broadcast on TV, a painting digitized on the WWW, a musical performance recorded and played on a phonograph), they would bear little narratological interest. But the shape of the pipe affects the kind of information that can be transmitted, alters the conditions of reception, and often leads to the creation of works tailor-made for the medium (cf. films made for TV). For the narratologist, channel-type media are only interesting to the extent that they involve “differences that make a narrative difference”—in other words, to the extent that they function as both conduits and “languages.” Among technologies, TV, radio, film, and the internet have clearly developed unique storytelling capabilities, but it would be hard to find reasons to regard Xerox copy machines or phonographs as possessing their own narrative “language.”
In Western thought, reflection on how narrative is conditioned by the medium in which it is realized—what we may call its mediality—can be traced as far back as Plato’s distinction between a diegetic and a mimetic mode of narration. According to Plato, in diegetic narration the poet speaks in his own voice (or rather, in the case of fiction, in the voice of a narrator), while in mimetic narration, he speaks through the characters. Both modes occur in epic poetry, but while diegetic narration, interpreted as reporting, remains dependent on language, mimetic narration, interpreted as showing, has become the dominant mode of presentation in multi-channel performing arts, such as drama, film, the opera, mime, and ballet. In these last two cases, as well as in silent film, mimetic narration becomes emancipated from language.
It was left to Aristotle to acknowledge medium as a distinctive property of art. After defining poetry as imitation (in the sense of representation), Aristotle mentions three ways of distinguishing various types of imitation: through medium, object and mode. Under medium, he classifies expressive resources such as color, shape, rhythm, melody, and voice. The notion of object (or content) creates a generic distinction between imitations that share the same medium: for instance, tragedy deals with people of higher standing, while comedy represents people of lower social stature. “Mode,” finally, covers Plato’s distinction between diegetic and mimetic presentation, but it is recast as an opposition between narration and impersonation: “It is possible to imitate the same objects in the same medium sometimes by narrating (either using a different persona, as in Homer’s poetry, or as the same person without variations), or else with all the imitators as agents and engaged in activity” (1996: 2.2). Here Aristotle regards narration and impersonation as instances of the same medium because both are made of language; but if we make a pragmatic distinction between enacting and reporting and regard this distinction as constitutive of medium, then their difference in “mode” marks epic poetry and drama as distinct narrative media in the modern sense of the word despite their common semiotic support.
Another landmark in the study of narrative mediality is Lessing’s distinction between spatial and temporal forms of art. Reacting to the 18th-century philosophy of art, which was captured by the saying of Simonides of Ceos, “painting is mute poetry, and poetry is speaking painting,” Lessing insisted on the sensory and spatio-temporal dimensions of the two media: painting speaks to the sense of sight, poetry to the imagination; painting extends in space, poetry extends in time. These differences predispose the two art forms to the representation of different subject matters: “signs existing in space can only represent objects whose wholes or parts coexist, while signs that follow one another can express only objects whose wholes or parts are consecutive” ([1766] 1984: 78). While the strength of painting lies in the representation of beauty, which resides in a relation between the parts of an object, poetry excels at the representation of action because action develops in time. Painting is in essence a descriptive medium, and poetry a narrative one. But Lessing does not exclude the possibility of stretching each medium in the direction of the other. Poetry can dramatize the evocation of static objects by transforming spatial vision into temporal action, as Homer does when he describes Juno’s chariot by recounting how Hebe put it together piece by piece. The spatial arts, conversely, can overcome their narrative deficiency by selecting a so-called “pregnant moment” that offers a window on the preceding and following actions. Lessing’s example is the famous Greek sculpture of Laocoön, which shows the Trojan priest and his sons in the last moments of a hopeless struggle against a sea serpent.
While we can extract observations relevant to what we now call medium in earlier periods, it wasn’t until the 20th century, when technological inventions such as photography, film, the phonograph, radio, and television expanded the repertory of channels of communication and means of representation that the concept of medium emerged as an autonomous topic of inquiry. McLuhan, an inspiring but somewhat mercurial thinker, popularized the concept with his characterization of media as “extension of man,” his claim that media are “forms that shape and reshape our perceptions,” and his oft-quoted but variably interpreted slogan “the medium is the message” (1996), which puts self-reference at the center of media studies. He was also instrumental in breaking down the barrier between elite and popular culture, a move which lead to the emancipation of media studies from literature, philosophy, and poetics. For McLuhan, comic strips, advertisements or the composition of the newspaper front page were as worthy of attention as works of high literature. But it was his disciple, Ong (1982), who broke the ground for the study of narrative in media other than written literature with a systematic explorations of the forms of narrative in oral and chirographic cultures (=cultures based on handwriting).
In France, the structuralist/semiotic movement gave legitimacy to the study of non-verbal forms of representation (advertisement and photography for Barthes [1980] 1981, cinema for Deleuze [1983] 1986, [1985] 1989 and Metz [1968] 1974, TV and mass communication for Baudrillard [1981] 1994). However, structuralism sometimes hampered the understanding of media due to its insistence on regarding Saussure’s linguistic theory as the model of all semiotic systems. Visual representations, in particular, cannot be divided into discrete units comparable to the morphemes and phonemes of language, and the doctrine of the arbitrariness of the linguistic sign cannot account for the iconic signification of painting and film. In the long run, Peircian semiotics, with its tripartite division of signs into symbols, icons and indices, has proved more fruitful for media studies.
The founding fathers of narratology recognized from the very beginning the medium-transcending nature of narrative: according to Bremond (1973), stories can be realized in media as diverse as literature, stage, ballet, and film. Mixing genres (Hühn & Sommer → Narration in Poetry and Drama) and media, Barthes ([1966] 1977) expands the list to include myth, legend, fable, tale, novella, epic history, drama, mime, painting, stained glass window, cinema, comics, news items, conversation, etc. Were he alive today, he would add blogs, hypertext, and video games. Barthes’ and Bremond’s wish to open up narratology to media other than literature went unfulfilled for years. Under the influence of Genette, narratology developed as a project almost exclusively devoted to literary fiction. Media representing the mimetic mode, such as drama and film, were largely ignored, and because of their absence of narrator, sometimes not even recognized as narratives, despite the similarity of their content with the plots of diegetic narration. But this situation changed dramatically in the late 20th century with the so-called “narrative turn” in the humanities. In the past twenty years, the study of non-literary or non-verbal forms of narrative has extended to conversational narrative (Labov 1972), film (Bordwell 1985; Chatman 1978), comic strips (McCloud 1994), painting (Bal 1991; Steiner [1988] 2004), photography (Hirsch 1997), opera (Hutcheon & Hutcheon 1999), television (Kozloff 1992; Thompson 2003), dance (Foster 1996), and music (Abbate 1989; Grabócz 1999, 2007: 231–298; Tarasti 2004; Seaton 2005).
Media studies took a theoretical turn in the 1990s. In the U.S., Bolter and Grusin (1999) proposed the concept of “remediation” to explain the relations between different media. In their view, every new technology-based medium must be understood, in the context of other media, as an attempt to “remediate” their limitations and get closer to the elusive goal of “achieving the real.” Video games, for instance, remediate film by incorporating narrative techniques commonly used in cinema within an interactive environment; digital photography remediates analogue photography by making images easier to manipulate; analogue photography remediates painting by being more faithful to its object; and the Internet remediates all other media by encoding them digitally in order to facilitate their transmission. In its narratological applications, “remediation” directs attention to how narrative texts may create networks of connections between different media. But the claim that every new medium constitutes an improvement over an old one cannot be sustained from a narratological and aesthetic point of view, for every gain in ex-presseness comes at a cost, and new media do not necessarily produce better narratives than old ones.
The concept of “intermediality,” now widely adopted in Europe, is more narrowly focused on art forms than remediation, and it avoids the meliorism inherent in this term. As Wolf (2008) observes, intermediality can be conceived in a narrow and in a broad sense. In a broad sense, it is the medial equivalent of intertextuality and covers any transgression of boundaries between different media. In a narrow sense, it refers to the participation of more than one medium—or sensory channel—in a given work. The opera, for instance, is intermedial through its use of gestures, language, music, and visual stage setting. If intermediality is interpreted in a wide sense, other terms must be forged to differentiate its diverse forms, including a new term for the narrow sense. Wolf (2005) suggests “plurimediality” for artistic objects that include many semiotic systems; “transmediality” for phenomena, such as narrative itself, whose manifestation is not bound to a particular medium; “intermedial transposition” for adaptations from one medium to another; and “intermedial reference” for texts that thematize other media (e.g. a novel devoted to the career of a painter or composer), quote them (insertion of text in a painting), describe them (representation of a painting through ekphrasis in a novel), or formally imitate them (a novel structured as a fugue).
In recent years, under the influence of Günther Kress and Theo van Leeuwen (2001), the term of “multimodality” has become established for works that combine several types of signs, such as images and text. In this new terminology, language, image and sound are regarded as “modes” rather than as “media,” as they would be when one adheres to the second of Webster’s definitions, but Kress and van Leeuwen’s terminology raises the problem of how to draw the line between modes and media.
The variety of the phenomena subsumed under the concept of medium stems not only from the two distinct functions mentioned by Webster’s definition—transmitting information or forming the support of information—but also from the nature of the criteria that differentiate individual media. These criteria belong to three conceptual domains: semiotic, material-technological, and cultural, each of which can be linked to different approaches to narrative.
As a semiotic category, a medium is characterized by the codes and sensory channels upon which it relies. The semiotic approach tends to distinguish three broad media families: verbal, visual, and aural. The groupings yielded by this taxonomy broadly correspond to art types, namely literature, painting, and music. This rudimentary typology must be expanded in order to account for an art like dance, which is based on the movements of the body, or for an activity like video games, whose distinctive feature is the pragmatic notion of active user participation. Insofar as signs extend in time or space, the semiotic analysis of media should also take into consideration their spatio-temporal dimensions. Media can be temporal and dynamic (music, oral language transmitted through radio or telephone), temporal and static (i.e. relying on sequentially ordered signs but freezing them through inscription, as in written literature); they can be purely spatial (painting, photography, sculpture, architecture) or spatio-temporal; the spatio-temporal in turn can be a static combination of temporal language and spatial image or inscription (comics, written literature that exploits the two-dimensionality of the page), or include a kinetic dimension that controls the duration of the receptive act (film, drama, mime, dance, and oral narrative accompanied by gestures). A semiotic approach to media focused on narrative will ask about the storytelling abilities and limitations of the signs of the medium under consideration. For instance: How can images suggest time? How can gestures express causality? What is the meaning of the graphic layout? How do the various types of signs contribute to narrative meaning in plurimedial art forms?
To bring further refinement to semiotic media families, we must ask about the material support of their individual members. Material support can be either a raw substance, such as clay for pottery, stone for sculpture, the human body for dance, and the human vocal apparatus for singing and oral storytelling, or a technological invention such as writing (subdivided into manuscript, print, and electronic form), individual musical instruments, photography, film, television, the telephone, and digital technology. (As a meta-medium that encodes all other media, digital technology would be a pure conduit, but by adding interactivity to these media, it reaches the status of “language.”) For the narratologist, the importance of technology lies in its ability to improve or modify the expressive power of purely semiotic media. A case in point is the well-documented and deep-reaching impact of the invention of writing, and later of print technology, on the form, use and content of narrative. According to Ong (1982), the influence of writing is felt in the rising and falling contour of the dramatic plot (for Western drama, even though performed orally, relies on a written text), in the development of psychologically complex characters, in the epistemological focus of the detective story, and in the self-referentiality of the postmodern novel.
Not all phenomena regarded as media can be distinguished on the basis of technological and semiotic properties alone. Newspapers, for instance, rely on the same semiotic dimensions and printing technology as books, but “the press” is widely regarded by sociologists as a medium in its own right because it fulfills a unique cultural role in the “media ecology.” It is also to cultural practice that we can attribute the grouping of semiotic dimensions into multi-channel media such as drama, the opera, and comic books, or, with the help of a technological support, into film, television, and computer games. The properties of narratives produced in a certain medium are often due to a combination of cultural, technological, and semiotic factors. The prevalence of shooting in American computer games could for instance be explained culturally by the importance of guns in American society (Japanese games are much less violent), as well as by the fact that the computer-game industry targets an audience of young males. But it is also motivated semiotically by the presence of a sound track (shooting is primarily manifested through noise) as well as technologically facilitated by the fact that the action of shooting is easily simulated by the manipulation of controls (hitting a key is reasonably similar to releasing a trigger). By far the majority of media studies have been devoted to the cultural use of medium-specific narratives. Possible topics for this approach include the rhetoric of TV news or the social impact of such phenomena as computer games, Internet pornography, and film violence.
Though we lack documents about the earliest manifestations of narrative among higher primates, it is reasonable to assume that language capacities, storytelling abilities, and human cultures co-evolved in symbiotic relation with each other. Dautenhahn (2003) attributes the need to tell stories to the complex social organizations of humans, compared to that of apes, while Turner (1996) argues that humans did not start telling stories as the result of developing language, but rather that language was developed in response to the need to tell stories. In these accounts of the social and cognitive foundations of storytelling, natural language is presented as the original narrative medium. The innate affinity of narrative and language can be explained by the fact that narrative is not something that is perceived by the senses: it is constructed by the mind, either out of data provided by life or out of invented materials. Similarly, as a mode of representation, language speaks to the mind rather than to the senses, though it is of course through the senses that its signs are perceived. Thanks to its semantic nature and its power of articulation, language is the only semiotic system (besides formal notation systems) in which it is possible to formulate propositions. Stories are about characters placed in a changing world, and narration is crucially dependent on the ability of a medium to single out existents and attribute properties to them. Neither images nor pure sound possesses this intrinsic ability: sound has no meaning, and pictures can show, but they cannot refer (Worth 1981). This makes it difficult for them to foreground specific properties of objects out of the background of their global visual appearance.
If we look at the constitutive features of narrative, we see other reasons why natural language is its medium of choice. Narrative is widely regarded by scholars as a discourse that conveys a story; story, in turn, has been defined as a mental image formed by four types of constituents (Ryan 2007): (1) a spatial constituent consisting of a world (the setting) populated by individuated existents (characters and objects); (2) a temporal constituent, by which this world undergoes significant changes caused by non-habitual events (Hühn → Event and Eventfulness); (3) a mental constituent, specifying that the events must involve intelligent agents who have a mental life and react emotionally to the states of the world (or to the mental states of other agents); (4) a formal and pragmatic constituent, advocating closure and a meaningful message.
The first and fourth of these conditions are not particularly dependent on language. Closure and meaningfulness can be achieved in any semiotic system, and images are more efficient than words at representing a world populated by existents because of the spatial extension and visual appearance of concrete objects. But the second and third features of narrative are highly language-dependent. As Lessing observed, the temporality of language is naturally suited to represent events that succeed each other in time. With its combination of dynamic unfolding and visuality, film may be as efficient as words at representing a succession of events such as “the king died and then the queen died,” but only words can say “the king died and then the queen died of grief” because only language is able to make relations of causality explicit. In a film (and even more so in a static image), causal relations between events must be left to the spectator’s interpretation, and without a voice-over narration (Kuhn & Schmidt → Narration in Film), we can never be completely sure that it was grief and not illness that killed the queen. Language-based narratives may admittedly choose to be highly elliptic in their presentation of causal relations: nothing would be more tedious than a story that left nothing to infer, but if all causal relations had to be guessed, this would place serious limitations on the repertory of stories that can be told by a medium. However, it is with condition 3 that language displays its true narrative superiority over other semiotic media. In language, we can express emotions and intents unambiguously by saying “x was scared,” “x was upset,” “x was in love,” or “x decided to take revenge.” Language can dwell at length on the mental life of characters, on their considerations of multiple possible courses of actions, on their philosophy of life, on their hopes and fears, on their daydreams and fantasies, because mental life can be represented as a kind of inner discourse, structured in the same way as language. Cognitive science may tell us that not all thinking is verbal, but the translation of private thought into language is one of the most powerful and widespread narrative devices. Most importantly, only language can represent the most common type of social interaction between intelligent agents, namely verbal exchanges, for the very simple reason that only language can represent language. The narrative power and diversity of film, drama and the opera is mainly due to the presence of a language track. This track, traditionally, has been limited by the conventions of realism to what an observer looking through an imaginary fourth wall can hear, namely dialogue. But phenomena such as the chorus of Greek tragedy, the written signs of epic theater, the asides to the audience of modern drama, and the voice-over narration of film represent an attempt to use language not only to imitate the speech of characters, but also to comment on the action, as it does so often in diegetic narrative. The storytelling potential of a medium is directly proportional to the importance and versatility of its language component.
The independence of narrative from language is a matter of degree. In its strictest interpretation, “narrating without language” means that a story unknown to the appreciator is evoked by the purely sensory, non-semantic resources of image or sound. (Taste, touch, and smell are far less developed senses, and they do not seem to have any narrative potential.) In a slightly weaker form of non-verbal narrativity (Abbott → Narrativity), the work tells a story new to the user, but it uses a language-based title to suggest a narrative interpretation. In the loosest interpretation, a narrative without language is a work that illustrates a story already known to the user (Varga 1988), and its narrativity is parasitic on the narrativity of the original text, which, most likely, will be known through language. This illustrative function is by far the most common occurrence in non-verbal narration.
To achieve narrativity, pictures must capture the temporal unfolding of a story through a static frame. Wolf (2005) distinguishes three kinds of pictorial narratives: monophase works that evoke one moment in a story through a single image; polyphase works that capture several distinct moments within the same image; and series of pictures that capture a sequence of events.
The monophase work presents the greatest narrative challenge because it must compress the entire narrative arc into a single scene. For an image to suggest a narrative interpretation, it must not only represent a frozen moment in a dynamic action, but must also arouse curiosity about the motivation of the agent. From very early on, the visual arts have shown man in action, but the hunting scenes or everyday activities depicted in cave paintings or on Egyptian scrolls do not fully qualify as narratives because they represent repetitive events with an unproblematic life-maintenance function. Similarly, the scenes of 17th-century Dutch genre painting are low in narrativity, or more specifically in eventfulness, because they rely almost entirely on familiar scripts and schemata for their interpretation. A truly narrative image must depict one-of-a-kind events that cause a significant change of state for the participants: not baking bread but stealing a loaf; not hunting animals for food, but killing a dragon to save a princess; not making music as a group, but secretly fondling a fellow musician (cf. Hühn’s distinction between event I and event II in the present encyclopedia (Hühn → Event and Eventfulness)). To read a picture narratively is to ask: Who are the characters shown in the picture? What are their interpersonal relations? What have they done before? What are they doing? What are their reasons for acting? What change of state will the action bring? How will the characters react to the event? Pictures cannot answer these questions directly because they are limited to the representation of visual properties. Not only do images lack a temporal dimension, they are also unable to represent language and thought, causal relations, counterfactuality, and multiple possibilities. Other limitations include the inability to make comments, provide explanations, and create suspense and surprise, two effects which depend on a time-bound disclosure of information. Even so, the narrative incompleteness of images is a powerful generator of curiosity. As Wolf (2005) has shown, reading a picture narratively necessitates a far more elaborate gap-filling activity than reading a language-based story. Monophase pictorial narratives are either illustrative or indeterminate in their content. An indeterminate picture opens a small window on time through the technique of the pregnant moment, but many different narrative arcs can pass through this window, corresponding to the multiple ways of imagining the long-term past and future that expand the content of the window into a complete story. Perhaps the only type of monophase pictures that tells a determinate story is the humorous single-frame, caption-less cartoon, for humor lies in a narrowly defined feature that people either get or miss.
Yet still pictures also have their narrative strengths, when compared to language: they can give a far better idea of the spatial configuration of the storyworld; they can suggest emotions through facial expressions and body language; and they can show beauty directly, rather than naming the property and leaving its specific representation to the reader’s imagination. Though they lack operators of mental activity, they can develop visual conventions, such as the thought balloon, to “derealize” events and represent objects as mental images formed by characters. They often make up for their inability to name characters by using traditional attributes (keys for Saint Peter, horns for the devil), and they can suggest abstract ideas through conventional visual symbols: lilies for purity, pomegranates for lust, a skull for death. When purely visual means fail, they can internalize language by showing intra-diegetic objects bearing inscriptions, such as signs or letters (cf. the very readable letter from Charlotte Corday held by the dead Marat in Jacques-Louis David’s “Marat Assassinated”). Because pictures stand still, the spectator has ample time to inspect them for narratively significant details.
In polyphase pictures, the narrative arc is much more determinate because it is plotted through several distinct scenes within the same global frame. These scenes are often separated by architectural features: for instance, in Benozzo Gozzoli’s “The Dance of Salome and the Beheading of St John the Baptist” (cf. Steiner [1988] 2004), an arched wall separates the beheading scene from the dancing scene, and Salome presents the head of the saint to her mother Herodiad in an alcove of the room where the dancing scene takes place. The space of the pictures may or may not be used as an indicator of temporal sequence: in “The Dance of Salome,” the eye does not read the story told by the painting linearly (i.e. left to right or right to left), but follows a circular path, from the right to the left to the center. This path must be discovered by detecting relations of causality which parallel the direction of time. But the narrative gaps between the individual scenes are so great in this particular painting that a spectator unfamiliar with the biblical story would be unable to decode its narrative logic. Themes such as reward and revenge, crucial to the Salome story, involve mental constructs far too complex for visual representation.
It takes a series of pictures to tell a story that is both reasonably determinate and new to the reader. Serial pictures can narrate in two ways. The first, illustrated by William Hogarth’s painting series A Rake’s Progress and Marriage à la Mode (Wolf 2005), consists of devoting each picture to one episode in the life of a character by resorting to the techniques of the monophase pictures. The individual paintings depict self-contained mini-narratives separated from each other by significant time gaps, but the various scenes are connected by weak causal relations: each painting represents a step in the downfall of the hero, a young man who rises from poverty through inheritance, engages in a life of debauch and dishonesty, gambles his fortune away, is imprisoned and ends up in a mental asylum. Narrative content is suggested on the level of the individual images by their reliance on familiar scripts, such as the gambling-house or the prison script, and on the global level by the recurrence of the same character (identified by constant visual features), as well as by the chronological sequence indicated by the spatial arrangement of the pictures. The other technique, common in wordless comic strips, associates every image with one moment in a continuous action as if it were a frozen frame in a silent film. While in the first technique narrativity exists on both the macro- and the micro-level, here it is limited to the macro-level. The individual images are separated by smaller time spans than in the first type, but they are linked together by stronger causal relations. An example of this technique is a sketchbook titled “Pipe Dreams” by the French artist Jean-Jacques Sempé, published in The New Yorker on November 20, 2000. “Pipe Dreams” tells the story of a lion who fantasizes loving a unicorn. But since unicorns do not exist, he marries a mare and tries unsuccessfully to turn her into a unicorn by putting an ice cream cone on her forehead. The upset bride runs away from him, and he ends up on a psychiatrist’s couch. Through the use of speech and thought balloons, the narrative is able to perform a rare feat in wordless storytelling: a disruption of the chronological order. After an opening frame that shows the lion dreaming of a unicorn, the next five frames (out of fourteen) represent the lion on the couch, and his personal experience is shown as images within a speech balloon, suggesting that it is being told to the psychiatrist. When the lion’s story escapes from the balloon and fills the entire frame, the storytelling act disappears from sight, and the reader is transported back to the time of the narrated events. The embedded sequence of the past catches up in the last frame with the embedding sequence of the present when the lion is shown knocking on the psychiatrist’s door. Thanks to the visual conventions of the modern comic strip, “Pipe Dreams” remediates many of the limitations of the purely mimetic image without using a single word: even the title is not indicative of narrative content.
As ballet, pantomime, and the movies of the silent area demonstrate, it is possible to tell a story through the kinetic means of gestures and facial expression. But ballet either fulfills an illustrative function (cf. for this aspect also 3.4.3 on music) with respect to the story referred to by its title (“Cinderella,” “The Nutcracker”) or relies on a summary in the program, while silent movies use music and subtitles to suggest a narrative interpretation. Can body movement tell a story that is new to the spectator without external help? The answer is yes, but the repertory is very limited. A pantomime could for instance tell the story of a scorned lover who becomes depressed and attempts suicide, but suddenly regains his lust for life when an attractive woman walks by. Narrative is about evolving networks of human relations; and gestures and movement, by varying the distance between bodies, are reasonably good at representing the evolution of interpersonal relations, as long as mental life can be translated into visible body language. But even though gestures add a kinetic element to serial still pictures, this does not result in a significant increase in narrative power. On the contrary: it is much more difficult to narrate through continuous gestures than it is through discrete pictures frames. The chronological rearrangements of the Sempé cartoon would be impossible in a pantomime because gestural narration unfolds entirely in the present. It also operates in a simulacrum of real time that largely limits the narrated time to the time of narration. This real time dimension predisposes gestural narration to the representation of short sketches. Serial pictures, by contrast, break the continuity of action into distinct frames, and the frames are separated by variable time spans: from a fraction of a second when cartoons reproduce continuous action to a lengthy period of time when frames introduce new episodes. Gestural narration could admittedly signal breaks between episodes by making the actors disappear from the stage and reappear. But in contrast to still pictures, language, and film, the live performance of gestural narration is incapable of skipping a moderate period of time. It is only when gestures are recorded through film and the footage put together through montage that it becomes possible to create ellipses of any length in the development of a narrative action (e.g. Bordwell & Thompson 2008: 229–231).
Music has a long history of being paired with language for narrative effects (sung poetry, “texted” music, opera, sound track of film and computer games), but it may seem paradoxical to even mention the possibility of telling stories through pure sounds. As a semiotic substance, sound possesses neither the conventional meaning nor the iconic value that allow words and images to create a concrete world and bring to mind individuated characters. Music cannot imitate speech, represent thought, narrate actions, or express causal relations. Its mimetic abilities are limited to the imitation of aural phenomena: the gurgling of a brook, the song of birds, or the rumbling of thunder. Yet in the 19th century, composers frequently attempted to tell stories through music by patterning their works according to what musicologists call a “narrative program.” These programs, expressed in words, instruct the listener’s imagination to look for a precise theme in each part of the composition: for instance, “Awakening of joyful sensations on arrival in the country” and “Scenes at a brook” as the titles of movements in Beethoven’s Pastoral symphony. More recently, a school of musicology has postulated the existence of a “deep narrativity” inherent to all music (or at least, to all music of the classical Western tradition). To tease out this deep narrativity, scholars resort to well-known narratological models such as Greimas’ semiotic square and Propp’s functions (Tarasti 2004), Ricœur’s theory of narrative temporality (Grabócz 1999), or the classical plot schema of equilibrium, conflict and resolution (Seaton 2005). Comparisons have also been made with diegetic and mimetic modes of storytelling (Abbate 1989), leading to the conclusion that music is a mimetic mode when it stands by itself, but fulfills a diegetic function when it is used in plurimedial works such as film and musicals (Rabinowitz 2004). In mimetic modes, according to the narrative school, music itself counts as narrative action, while in diegetic modes, it comments upon the enacted events.
The appeal of the concept of narrative to both composers and musicologists can be explained by the temporal dimension of music. Narrative lives from a succession of events that brings transformations to the state of the storyworld, while music lives from a succession of sounds that creates melody and harmony through transformations in pitch, rhythm, and loudness. The term “line” is used to describe the development of both plot and melody, and in each case, this line controls attention, builds expectations, and creates effects of suspense, curiosity, and surprise (Sternberg 1992). But unlike verbal narrative, music does not suggest the passing of time by showing its effects on concrete existents: it captures time in its pure form, as a forward movement, a desire-for-something-to-come, a tension calling for a resolution. In music as in narrative, the appreciator may have a powerful sense that a dénouement is imminent (perhaps more so in music, for in literature the coming end is often signaled not by narrative devices, but by the number of pages left to be read). Through its modest descriptive abilities, music can sometimes sketch a setting (cf. Beethoven’s Pastoral symphony), and in what amounts to creating its own conventional “language,” it can individuate characters by linking them to a specific instrument or to a leitmotiv. It also possesses an ability unequalled among semiotic media to represent and induce emotions. But these features are not sufficient to tell specific stories. In contrast to the narrativity of language-based texts, the narrativity of music is neither determinate nor literal. It is indeterminate because narrative content is something that is read into a composition rather than read from it (Wolf 2005). Even when music instructs the listener to associate the composition with a certain story, every listener fills in the general pattern in a highly personal way (Nattiez 1990), and many listeners will appreciate the composition without giving any thought to a narrative interpretation. This would be unthinkable with a language-based story. Meanwhile, from the point of view of the musicologist who uses narratological models to analyze particular compositions, the alleged narrativity of music is the product of a metaphor based on a structural analogy. Music and language-based stories present similar formal patterns, but these patterns are filled with vastly different substance: intrinsically meaningless sound in the case of music (though of course musical arrangement creates its own type of meaning), concrete semantic content in the case of language-based stories. As the focus of interest of a scholarly approach, the narrativity of music is a purely analytical construct situated, cognitively, on a very different level than the narrativity of language, film, or even pictures because it can exercise its power without being consciously recognized.
Given the overwhelming storytelling superiority of language, one may wonder why mankind ever bothered to develop other narrative media. The limited narrative power of non-verbal media does not mean, however, that they cannot make original contributions to the formation of narrative meaning. The affordances of language, pictures, movement, and music complement each other, and when they are used together in multi-modal media, each of them builds a different facet of the total imaginative experience: language narrates through its logic and its ability to model the human mind, pictures through their immersive spatiality and visuality, movement through its dynamic temporality, and music through its atmosphere-creating, tension building and emotional power.
The ultimate goal of art is to involve the whole of the embodied mind, the intellect as well as the senses. To achieve this wholeness, sensorial art forms must be coaxed into conveying messages, while language-based art forms must be taught to appeal to the senses. Through narrativization, sensorial arts acquire a sharper mental dimension, and through collaboration with sensorial signs, language-based narrative allows a fuller experience of the storyworld. In multi-modal media, the appreciator can directly see, hear, and maybe even interact with objects, and the imagination, relieved from the cognitive burden of simulating sensory data, can more easily immerse itself in the story. But this does not mean that multi-modal media are automatically superior to literature in narrative power because every gain in the visual, aural or even interactive domain may bring a loss of attention to the language channel (e.g. for the relation between audiovisual and voice-over narration in film Kozloff 1988: 8–22).
Research concerning the relations between media and narrative has recently taken two major directions. The first is an increased interest in multimodality. Narrative forms combining a variety of semiotic channels have existed since the dawn of civilization, if one thinks of the inherent multimodality of oral storytelling (voice + gestures), but every new technology of communication inspires novel combinations: printing allowed the wide distribution of illustrated books, and later of comics; photography gave birth to photonovels; cinema integrated animated images, music, spoken language and occasionally written text, and digital technology added interactivity to the many modes of film. The current interest in multimodality has led to a reappraisal of some easily overlooked modes, such as the gestures of oral storytelling, the sound track of film, or the choreography of actor movements on stage. It has also focused attention on the increasingly common insertion within novels of a variety of non-verbal documents, such as photos, handwritten notes, graphs, and maps (Hallett 2014)—a list that could extend to video clips and animation for Web-based texts.
The other new research area focuses on the spreading of narrative content across multiple media platforms. Widely known as “transmedia” or “transmedial” narration, and first described by Henry Jenkins (2006; see also Dena 2009 and Mittell 2014), this important trend in contemporary culture comes in two types. The first could be called the “snowball effect”: a certain story enjoys so much popularity, or becomes culturally so prominent, that it spontaneously generates a variety of either same-medium or cross-media prequels, sequels, fan fiction and adaptations. In this case there is a central text that functions as common reference for all the other texts. Harry Potter and Lord of the Rings are good examples of the snowball effect: they started out in the medium of the novel, created by a single author, and they expanded to film and computer games by popular demand. In the other type of transmedial narration, illustrated by the commercial “franchise” of The Matrix, which comprises films, computer games and comics, a certain story is conceived from the very beginning as a project that develops over many different media platforms (Ryan 2013). The phenomenon of transmedial storytelling raises important theoretical questions, such as: are the component of the system autonomous, or do they presuppose knowledge of other members of the network; how do the storyworlds of the various texts relate to each other (i.e. can they be regarded as regions of the same global world or are they logically incompatible?); what elements must be present for audiences to assume that, despite additions or modifications, texts of different media refer to the same storyworld; and what kind of stories inspire transmedial developments.
(1) What is the range of applicability of narratological concepts with respect to media (i.e. which ones apply to all media capable of narrativity, which ones are medium-specific, and which ones can be used for several media but not for all of them?) (2) How are certain constitutive dimensions of story, such as subjectivity, temporal sequence, or causality, or certain discourse strategies, such as focalization, represented in non-verbal media (Thon 2014; Horstkotte & Pedri 2011); (3) How do newly developed media progressively free themselves from the influence of older media and discover their own narrative “language”? (4) What social practices are generated by the “cult narratives” of mass media (e.g. practices such as the creation of fan communities on the Internet, fan fiction, spoiling, online discussions of plots)? (5) In which media, besides language, does fictionality exist? (6) What forms does (or will) narrative take in interactive environments?
Abbate, Carolyn (1989). “What the Sorcerer Said.” 19th-Century Music 12, 221–230.
Aristotle (1996). Poetics. Tr. & intr. M. Heath. London: Penguin Books.
Bal, Mieke (1991). Reading Rembrandt: Beyond the Word-Image Opposition. New York: Cambridge UP.
Barthes, Roland ([1966] 1977). “Introduction to the Structural Analysis of Narrative.” Image Music Text. New York: Hill & Wang, 79–124.
– ([1980] 1981). Camera Lucida. Reflections on Photography. New York: Hill & Wang.
Baudrillard, Jean ([1981] 1994). Simulacra and Simulations. Ann Arbor: U of Michigan P.
Bolter, Jay David & Richard Grusin (1999). Remediation: Understanding New Media. Cambridge: MIT P.
Bordwell, David (1985). Narrative in the Fiction Film. Madison: U of Wisconsin P.
– & Kristin Thompson (2008). Film Art. An Introduction. New York: McGraw-Hill.
Bremond, Claude (1973). Logique du récit. Paris: Seuil.
Chatman, Seymour (1978). Story and Discourse. Narrative Structure in Fiction and Film. Ithaca: Cornell UP.
Dautenhahn, Kirsten (2003). “Stories of Lemurs and Robots: The Social Origin of Story-Telling.” M. Mateas & Ph. Sengers (eds.). Narrative Intelligence. Amsterdam: Benjamins, 63–90; also on WWW at <http://homepages.feis.herts.ac.uk/~com-qkd/kdnarrative.pdf>.
Deleuze, Gilles ([1983] 1986). Cinema 1: The Movement Image. Minneapolis: U of Minnesota P.
– ([1985] 1989). Cinema 2: The Time-Image. Minneapolis: U of Minnesota P.
Dena, Christy (2009) Transmedia Fictions: Theorising the Practice of Expressing a Fictional World across Distinct Media and Enviroments. Ph.D. Dissertation, University of Sidney.
Foster, Susan Leigh (1996). Choreography and Narrative. Bloomington: U of Indiana P.
Grabócz, Márta (1999). “Paul Ricœur’s Theories of Narrative and Their Relevance for Musical Narrativity.” Indiana Theory Review 20, 19–40.
– (2007). Sens et signification en musique. Paris: Hermann.
Hallet, Wolfgang (2014). “The Rise of the Multimodal Novel: Generic Change and ist Narratological Implications.” M.-L. Ryan & J.-N. Thon (eds.). Storyworlds Across Media. Lincoln: U of Nebraska P.
Hirsch, Marianne (1997). Family Frames: Photography, Narrative, and Postmemory. Cambridge: Harvard UP.
Horstkotte, Silke & Nancy Pedri (2011). “Focalization in Graphic Narrative.” Narrative 19.3, 330–357.
Hutcheon, Linda & Michael Hutcheon (1999). Opera: Desire, Disease, Death. Lincoln: U of Nebraska P.
Jenkins, Henry (2006). Convergence Culture: Where Old and New Media Collide. New York: New York UP.
Kozloff, Sarah (1988). Invisible Storytellers. Voice-Over Narration in American Fiction Film. Berkeley: U of California P.
– (1992). “Narrative Theory and Television.” R. C. Allen (ed.). Channels of Discourse, Reassembled. Chapel Hill: U of North Carolina P, 43–71.
Kress, Günther & Theo van Leeuwen (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.
Labov, William (1972). Language in the Inner City. Studies in the Black English Vernacular. Philadelphia: U of Pennsylvania P.
Lessing, Gotthold Ephraim ([1766] 1984). Laocoön: An Essay on the Limits of Painting and Poetry. Baltimore: Johns Hopkins UP.
McCloud, Scott (1994). Understanding Comics. New York: Harper Perennials.
McLuhan, Marshall (1996). E. McLuhan & F. Zingrone (eds.). Essential McLuhan. New York: Basic Books.
Metz, Christian ([1968] 1974). Film Language. A Semiotics of the Cinema. New York: Oxford UP.
Meyrowitz, Joshua (1993). “Images of Media: Hidden Ferment—and Harmony—in the Field.” Journal of Communications 43, 55–66.
Mittell, Jason (2014). “Strategies of Storytelling on Transmedia Television.” M.-L. Ryan & J.-N. Thon (eds.). Storyworlds Across Media. Lincoln: U of Nebraska P.
Ong, Walter J. (1982). Orality and Literacy. The Technologizing of the Word. London: Methuen.
Nattiez, Jean-Jacques (1990). “Can one Speak of Narrativity in Music?” Journal of the Royal Musical Association 115, 240–257.
Rabinowitz, Peter (2004). “Music, Genre, and Narrative Theory.” M.-L. Ryan (ed.). Narrative across Media: The Languages of Storytelling. Lincoln: U of Nebraska P, 305–328.
Ryan, Marie-Laure (2007). “Toward a Definition of Narrative.” D. Herman (ed.). The Cambridge Companion to Narrative. Cambridge: Cambridge UP, 22–35.
– (2013). “Transmedial Storytelling and Transfictionality.” Poetics Today 34.3, 361–388.
Seaton, Douglas (2005). “Narrative in Music: The Case of Beethoven’s ‘Tempest’ Sonata.” J. Ch. Meister (ed.). Narratology beyond Literary Criticism. Mediality, Disciplinarity. Berlin: de Gruyter, 65–82.
Steiner, Wendy ([1988] 2004). “Pictorial Narrativity.” M.-L. Ryan (ed.). Narrative across Media: The Languages of Storytelling. Lincoln: U of Nebraska P, 145–177.
Sternberg, Meir (1992). “Telling in Time (II): Chronology, Teleology, Narrativity.” Poetics Today 13, 463–541.
Tarasti, Eero (2004). “Music as Narrative Art.” M.-L. Ryan (ed.). Narrative across Media: The Languages of Storytelling. Lincoln: U of Nebraska P, 283–304.
Thompson, Kristin (2003). Storytelling in Film and Television. Cambridge: Harvard UP.
Thon, Jan-Noël (forthcoming). “Subjectivity across Media: On Transmedial Strategies of Subjective Representation in Contemporary Graphic Novels, Feature Films, and Computer Games.” M.-L. Ryan & J.-N. Thon (eds.). Storyworlds Across Media. Lincoln: U of Nebraska P.
Turner, Mark (1996). The Literary Mind. New York: Oxford UP.
Varga, A. Kibédi (1988). “Stories Told by Pictures.” Style 22, 194–208.
Wolf, Werner (2005). “Intermediality”; “Music and Narrative”; and “Pictorial Narrativity.” D. Herman et al. (eds.). The Routledge Encyclopedia of Narrative Theory. London: Routledge, 252–256, 324–329, and 431–435.
– (2008). “The Relevance of ‘Mediality’ and ‘Intermediality’ to Academic Studies of English Literature.” M. Heusser et al. (eds.). Mediality / Intermediality. Göttingen: Narr, 15–43.
Worth, Sol (1981). “Pictures Can’t Say Ain’t.” S. W. Studying Visual Communication. Philadelphia: U of Pennsylvania P, 162–184.
Kafalenos, Emma (2001). “Reading Visual Art, Making—and Forgetting—Fabulas.” Narrative 9.2, 138–145.
– (2004). “Overview of the Music and Narrative Field.” M.-L. Ryan (ed.). Narrative across Media: The Languages of Storytelling. Lincoln: U of Nebraska P, 275–282.
Nünning, Vera & Ansgar Nünning, eds. (2002). Erzähltheorie transgenerisch, intermedial, interdisziplinär. Trier: WVT.
Ryan, Marie-Laure (2005). “On the Theoretical Foundation of Transmedial Narratology.” J. Ch. Meister (ed.). Narratology Beyond Literary Criticism. Mediality, Disciplinarity. Berlin: de Gruyter, 1–23.
– (2006). Avatars of Story. Minneapolis: U of Minnesota P.
– & J.-N. Thon, eds. (2014). Storyworlds Across Media. Lincoln: U of Nebraska P.
Wolf, Werner (2002). “Das Problem der Narrativität in Literatur, Bildender Kunst und Musik: Ein Beitrag zu einer Intermedialen Erzähltheorie.” V. Nünning & A. Nünning (eds.). Erzähltheorie transgenerisch, intermedial, interdisziplinär. Trier: WVT, 23–104.