I first drew the Chart in order to clear up my own ideas on the subject, finding it very troublesome to retain a distinct notion of the changes that had taken place. I found it answer the purpose beyond my expectation, by bringing into one view the result of details that are dispersed over a very wide and intricate field of universal history; facts sometimes connected with each other, sometimes not, and always requiring reflection each time they were referred to.
—William Playfair, An Inquiry into the Permanent Causes of the Decline and Fall of Powerful and Wealthy Nations, 1805, in reference to “The Chart, No. 1, representing the rise and fall of all nations or countries, that have been particularly distinguished for wealth or power, is the first of the sort that ever was engraved, and has, therefore, not yet met with public approbation.”1
The pretty photographs we and other tourists made in Las Vegas are not enough. How do you distort these to draw a meaning for a designer? How do you differentiate on a plan between form that is to be specifically built as shown and that which is, within constraints, allowed to happen? How do you represent the Strip as perceived by Mr. A. rather than as a piece of geometry? How do you show quality of light—or qualities of form—in a plan at 1 inch to 100 feet? How do you show fluxes and flows, or seasonal variation, or change with time?
—Robert Venturi, Stefan Izenour, and Denise Scott Brown, Learning from Las Vegas: The Forgotten Symbolism of Architectural Form, 19722
“Whole” is now nothing more than a provisional visualization which can be modified and reversed at will, by moving back to the individual components, and then looking for yet other tools to regroup the same elements into alternative assemblages.
—Bruno Latour, “Tarde’s Idea of Quantification,” in The Social after Gabriel Tarde: Debates and Assessments, 20103
Information visualization is becoming more than a set of tools, technologies and techniques for large data sets. It is emerging as a medium in its own right, with a wide range of expressive potential.
—Eric Rodenbeck, keynote lecture at O’Reilly Emerging Technology conference, March 4, 20084
Visualization is ready to be a mass medium.
—Fernanda B. Viégas and Martin Wattenberg, “Interview: Fernanda Viégas and Martin Wattenberg from Flowing Media,” 20105
In 2000, only specialists in a few professional fields knew about information visualization. Ten years later, this changed completely: In 2010, the Museum of Modern Art in New York presented a dynamic visualization of its collection on five screens created by Imaginary Forces. The New York Times regularly featured custom visualizations both in its print and web editions, created by its in-house interactive team. Numerous sophisticated visualization projects created by scientists, designers, artists, and students appeared online. If you searched for certain types of public data, the first result returned by Google linked to an automatically created interactive graph of this data.6 If you wanted to visualize your own dataset, there were dozens of free online visualization tools and platforms such as Google Docs, Tableau Public, Plotly, and others. Three hundred years after William Playfair’s amazement at the cognitive power of information visualization, it looks like many others finally are getting it.
(A note on my use of terms: Today the terms information visualization and data visualization are often used interchangeably. Historically the first term preceded the second. To emphasize that visualization has a long history and was not magically born only recently, I decided to use “information visualization” in this chapter.)
I can’t address all aspects of information visualization as a new medium of visual communication and exploration in a single chapter. Nor can I teach you how to visualize your cultural datasets. There are numerous online tutorials, courses, and books that cover the craft of visualization, and you can learn from them and keep practicing until you achieve the desired results. Instead, this chapter focuses on a few ideas fundamental to visualization’s identity as a medium and its history: visualization as a mapping from one domain to another; visualization as a reduction of information; and visualization as a predominantly spatial representation. I then identify a new paradigm that I call media visualization. Whereas traditional information visualization demands reduction—using points, lines, and other geometric elements to stand in for real-world objects and their relations—media visualization operates without such reduction. I discuss a few well-known experimental projects to illustrate media visualization possibilities and explain the relevance of this paradigm for explorations of large cultural visual datasets.
What is information visualization? Despite the popularity of infovis (a common shortening of information visualization), it is not so easy to come up with a definition that will work for all kinds of visualization projects being created today and at the same will clearly separate them from those of other related fields, such as scientific visualization and information design. Let’s start with a provisional definition that we can modify later. We can define information visualization as a mapping between data and a visual representation. We can also use different concepts besides representation, each bringing an additional meaning. For example, if we believe that a brain uses a number of distinct representational and cognitive modalities, we can define infovis as a mapping from other cognitive modalities (such as mathematical and propositional) to an image modality.
This definition does not cover all aspects of information visualization—such as the distinctions among static, dynamic (i.e., animated), and interactive visualizations—the latter, of course, being the most important today. In fact, most definitions of infovis by computer science researchers equate it with the use of interactive, computer-driven visual representations and interfaces. Here are two examples of such definitions: “Information visualization (InfoVis) is the communication of abstract data through the use of interactive visual interfaces,”7 and “information visualization utilizes computer graphics and interaction to assist humans in solving problems.”8
Interactive graphic interfaces in general, and interactive visualization applications in particular offer all kinds of new techniques for manipulating data elements—from the ability to change how files are shown on the desktop in a modern OS to the multiple coordinated views made available in some visualization software such as Mondrian.9 However, regardless of whether you are looking at a visualization printed on paper or a dynamic interactive arrangement of graphic elements on your computer screen that you can change at any moment, in both cases this image is a result of mapping. So what is special about the images that such mapping produces?
For some researchers, information visualization is distinct from scientific visualization in that the latter uses numerical data, while the former uses non-numerical data such as text and networks of relations.10 Personally, I am not sure that this distinction holds in practice. Certainly, plenty of infovis projects use numbers as their primary data, but even when they focus on other data types, they still often use some numerical data as well. For instance, a typical network visualization may use both data about the structure of the network (which nodes are connected to each other) and quantitative data about the strength of these connections (e.g., how many messages are exchanged between members of a social network). As a concrete example of infovis that combines non-numerical and numerical data, consider History Flow ((Fernanda B. Viégas and Martin Wattenberg, 2003), one of the first projects to visualize large cultural data.11 This project shows how a given Wikipedia page grows over time as different authors contribute to it. The contribution of each author is represented by a line. The width of the line changes over time, reflecting the amount of text contributed by an author to the Wikipedia page. To take another infovis classic, Flight Patterns (Aaron Koblin, 2005) uses numerical data about flight schedules and trajectories for all planes that fly over the United States to create an animated map displaying the patterns formed by their movement over a twenty-four-hour period.12
Rather than trying to separate information visualization and scientific visualization using some a priori idea, let’s instead enter each phrase in a Google image search and compare the results. The majority of images returned by searching for information visualization are two-dimensional and use vector graphics—points, lines, curves, and other simple geometric shapes. The majority of images returned when searching for scientific visualization are three-dimensional; they use solid 3-D shapes or volumes made from 3-D points. The results returned by these searches demonstrate that the two fields indeed differ—not because they necessarily use different types of data but because they privilege different visual techniques and technologies.
Scientific visualization and information visualization come from different cultures (science and design); their development corresponds to different areas and eras of computer graphics technology. Scientific visualization developed in the 1980s along with the field of 3-D computer graphics, which at that time required specialized graphics workstations. Information visualization developed in the 1990s along with the rise of desktop 2-D graphics software and the adoption of PCs by designers; its popularity accelerated in the 2000s. The two key forces behind this acceleration were the easy availability of big datasets via APIs provided by major social network services from 2005 on, and new high-level programming languages specifically designed for graphics (i.e., Processing13) and software libraries for visualization (for instance, d314 and ggplot215).
Can we differentiate information visualization from information design? This is trickier, but here is my way of doing it: information design starts with data that already has a clear structure, and its goal is to express this structure visually. For example, the famous London tube map designed in 1931 by Harry Beck uses structured data: tube lines, stations, and their locations over London geography.16 In contrast, the goal of information visualization is to discover the structure of a (typically large) dataset. There is no a priori knowledge of this structure; a visualization is successful if it reveals this structure. A different way to express this is by saying that information design works with information, whereas information visualization works with data. As is always the case with actual cultural practice, it is easy to find examples that do not fit such a distinction—but a majority do. Therefore, I think that this distinction can be useful in allowing us to understand the practices of information visualization and information design as partially overlapping but ultimately different in terms of their functions.
Finally, what about the earlier practices of visual display of quantitative information in the nineteenth and twentieth centuries that are known to many via the examples collected in the pioneering books by Edward Tufte?17 Do they constitute infovis as we understand it today? As I already noted, most definitions provided by researchers working within computer science equate information visualization with the use of interactive computer graphics.18 Using software, we can visualize much larger datasets than was previously possible; create animated visualizations; show how processes unfold in time; share visualizations with others; and, most importantly, manipulate visualizations interactively. These differences are very important—but for the purposes of this chapter, which is concerned with the visual language of information visualization, they are less important. When we switched from pencils to computers, this did not affect the core idea of visualization: mapping some properties of the data into a visual representation. Similarly, while the availability of computers led to the development of new visualization techniques (scatterplot matrix, tree maps, various new types of network diagrams, etc.), the basic visual language of infovis remained the same as it was in the nineteenth century—points, lines, rectangles, and other graphic primitives. Given this continuity, I am using the term infovis to refer to both earlier visual representations of data created manually and contemporary, software-driven visualizations.
In my view, the practice of information visualization from its beginnings in the second part of the eighteenth century until today relied on two key principles. The first principle is reduction. Infovis uses graphical primitives such as points, straight lines, curves, and simple geometric shapes to stand in for objects and relations between them—regardless of whether these are people, their social relations, stock prices, income of nations, unemployment statistics, or anything else. By employing graphical primitives (or, to use the language of contemporary digital media, vector graphics), infovis is able to reveal patterns and structures in the data objects. However, the price being paid for this power is extreme reduction. We throw away 99 percent of what is specific about each object to represent only 1 percent in the hope of revealing patterns across this 1 percent of objects’ characteristics.
Information visualization is not unique in relying on such extreme reduction of the world to gain new power over what is extracted from it. It comes into its own in the first part of the nineteenth century when in the course of just a few decades almost all graph types commonly found today in statistical and charting programs were invented.19 This development of new techniques for visual reduction parallels the reductionist trajectory of modern science in the nineteenth century. Physics, chemistry, biology, linguistics, psychology, and sociology propose that both the natural and the social world should be understood in terms of simple elements—molecules, atoms, phonemes, just noticeable sensory differences, and the like—and the rules of their interaction. This reductionism becomes the default metaparadigm of modern science and it continues to rule scientific research today. For instance, currently popular paradigms of complexity and artificial life focus our attention on how complex structures and behavior emerge out of the interaction of simple elements.
Even more direct is the link between the nineteenth-century infovis and the rise of social statistics. Philip Ball summarizes the beginnings of statistics in this way: “In 1749 the German scholar Gottfried Achenwall suggested that since this ‘science’ [the study of society by counting] dealt with the natural ‘states’ of society, it should be called Statistik. John Sinclair, a Scottish Presbyterian minister, liked the term well enough to introduce it into the English language in his epic Statistical Account of Scotland, the first of the twenty-one volumes of which appeared in 1791. The purveyors of this discipline were not mathematicians, however, nor barely ‘scientists’ either; they were tabulators of numbers, and they called themselves ‘statists.’”20
In the first part of the nineteenth century, many scholars, including Adolphe Quetelet, Florence Nightingale, Thomas Buckle, and Francis Galton, used statistics to look for the “laws of society.” This inevitably involved summarization and reduction—calculating the totals and averages of the collected numbers about citizens’ demographic characteristics, comparing the averages for different geographical regions, asking if they followed a bell-shaped distribution, and so forth. It is therefore not surprising that many—if not most—graphical methods that are standard today were developed during this time for the purposes of representation of such summarized data. According to Michael Friendly and Daniel J. Denis, between 1800 and 1850, “in statistical graphics, all of the modern forms of data display were invented: bar and pie charts, histograms, line graphs and time-series plots, contour plots, and so forth.”21
Do all these different visualization techniques have something in common besides reduction? They all use spatial variables—position, size, shape, and, more recently, curvature of lines and movement—to represent key differences in data and reveal the most important patterns and relations. This is the second (after reduction) core principle of infovis practice as it was practiced for three hundred years—from the very first line graphs (1711), bar charts (1786), and pie charts (1801) to their ubiquity today in all spreadsheet software such as Excel, Numbers, Google Docs, OpenOffice, and the like.22
Infovis privileges spatial dimensions over other visual dimensions. In other words, we map the properties of our data in which we are most interested into topology and geometry. Other less important properties of the objects are represented through different visual dimensions—the tones, fill-in patterns, colors, transparency, size, or shape of the graphical elements.
As examples, consider two common graph types: bar chart and line graph. Both first appeared in William Playfair’s Commercial and Political Atlas, published in 1786, and became commonplace in the early nineteenth century. A bar chart represents the differences between data objects via rectangles that have the same width but different heights. A line graph represents changes in the data values over time via the changing height of the line.
Another common graph type—a scatter plot—similarly uses spatial variables (positions and distances between points) to make sense of the data. If some points form a cluster, this implies that the corresponding data objects have something in common; if you observe two distinct clusters, this implies that the objects fall into two different classes; and so on.
Consider another example—network visualizations that function today as distinct symbols of “network society” (see Manuel Lima’s authoritative gallery at visualcomplexity.com, which presents one thousand network visualization projects). Like bar charts and line graphs, network visualizations also privilege spatial dimensions: position, size, and shape. Their key addition is the use of straight or curved lines to show connections between data objects. For example, in Distellamap (2005), Ben Fry connects pieces of code and data by lines to show the dynamics of the software execution in Atari 2600 games.23 In Marcos Weskamp’s Flickr Graph (2005), the lines visualize the connections among users of Flickr.24 (Of course, many other visual techniques can also be used in addition to lines to show relations; see, for instance, a number of maps of science created by Katy Borner and her colleagues at the Information Visualization Lab at Indiana University.25)
I believe that the majority of information visualization practices from the second part of the eighteenth century until today follow the same principle—reserving the spatial variables for the most important dimensions of the data and using other visual variables for the remaining dimensions. This principle can be found in visualizations ranging from the famous dense graphic showing Napoleon’s march on Moscow by Charles Joseph Minard (1869)26 to the more recent The Evolution of the Origin of Species by Stefanie Posavec and Greg McInerny (2009).27 Distances between elements and their positions, shape, size, line curvature, and other spatial variables code quantitative differences between objects and/or their relations (e.g., who is connected to whom in a social network).
When visualizations use colors, fill-in patterns, or different saturation levels, typically this is done to partition graphic elements into groups. In other words, these nonspatial variables function as group labels. For example, Google Trends uses line graphs to compare search volumes for different words or phrases; each line is rendered in a different color.28 However, the same visualization could have simply used labels attached to the lines—without different colors. In this case, color adds readability, but it does not add new information to the visualization.
The privileging of spatial over other visual dimensions was also true of plastic arts in Europe between the sixteenth and nineteenth centuries. A painter first worked out the composition for a new work in many sketches; next, the composition was transferred to a canvas; and then shading was fully developed in monochrome. Only after that was color added. This practice assumed that the meaning and emotional impact of an image depends most of all on the spatial arrangements of its parts, as opposed to colors, textures, and other visual parameters. In classical Asian ink and wash painting, which first appeared in the seventh century in China and was later introduced to Korea and then Japan (in the fourteenth century), color did not even appear. Painters used exclusively black ink to explore the contrasts between objects’ contours, their spatial arrangements, and different types of brushstrokes.
It is possible to find information visualizations in which the main dimension is color—for instance, a common traffic light that “visualizes” the three possible behaviors of a car driver: stop, get ready, go. This example shows that if we fix spatial parameters of visualization, color can become the salient dimension. In other words, it is crucial that the three lights have exactly the same shape and size. Apparently, if all elements of the visualization have the same values on spatial dimensions, our visual system can focus on the differences represented by colors or other nonspatial variables.
Why do visualization designers—be they the inventors of graphing techniques at the end of the eighteenth and early nineteenth centuries or the millions of people who now use these graph types in their reports and presentations or the contemporary authors of more experimental visualizations featured in museum exhibitions—privilege spatial variables over other kinds of visual mappings? Why are color, tone, transparency, and shape used to represent secondary aspects of data while the spatial variables are reserved for the most important dimensions?
The answer comes from studies of vision in experimental psychology. Visualization designers exploit the strengths of the human visual system. Human vision can perceive very well small differences in spatial arrangements of elements and very quickly compare the sizes, directions, orientations, and shapes of these elements. Consequently, most popular visualization techniques rely on our ability to see these differences in positions of points, directions and curvature of lines, and relative sizes of bars. This ability is used to interpret scatter plots, line plots, and bar plots.
In comparison, human vision’s abilities to make quick comparisons among color hues, brightness levels, or levels of transparency are more limited. This is why visualization designers typically use these variables for secondary aspects of data. For example, you can assign around eight category values to different hues or tones. But if you try to do so for a larger number of values, it becomes more difficult to see patterns in a visualization. Similarly, if you want to represent values of a quantitative variable by using differences in color, tone, or transparency, you can do it successfully for a few dozen of values and sometimes maybe even a hundred, but not for a larger number. If you try to do this, you push beyond what human vision can easily process.
Why has the human visual system evolved to have superior spatial abilities? Why is the geometric arrangement of elements in a scene more important to human perception than other visual dimensions? Possibly this has to do with the fact that each object in the real world occupies a unique part of the 3-D space. Therefore, it is important for a human brain to be able to segment a visual field into spatially distinct objects that are likely to have different behaviors and uses for a human being—for example, animals, trees, fruits, or people; one animal versus a few; and so on. Object recognition also can benefit from an ability to register differences in structures of objects. Different types of objects have parts with different shapes. A tree has a trunk and branches; a human being has a head, a torso, arms, and legs. Identifying 2-D forms and their arrangements and registering small differences in size, shape, and orientation is thus likely to play an important role in object recognition. Yet another possible reason is the need to read facial expressions and recognize faces, which also requires high-resolution spatial perception. A face has eyes, nose, and mouth, plus other parts, and we need to be able to perceive tiny differences in their shapes, positions, and details.
A craftsperson, designer, or artist can create objects and compositions that focus our attention on nonspatial visual dimensions such as textures, colors or shades of one color, or reflective qualities of materials: think of ornaments in traditional human cultures, Matisse’s paintings, fashion designs by Missoni, the minimalist spaces that became popular in the 1990s, or phone designs in the 2010s with surfaces that use subtle colors and levels of reflectivity. But in everyday perception, spatial dimensions are what matter most. How close two people are to each other and their body positions; the expressions on their faces; their relative size, which allows us to estimate their distance from us; the characteristic shapes of different objects, which allows us to recognize them—all these and many other spatial characteristics that our brains instantly compute from retinal input are crucial for our daily existence.
This privileging of spatial variables in human perception may be the reason that all standard techniques for making graphs and charts developed in the eighteenth to twentieth centuries use spatial dimensions to represent key aspects of the data and reserve other visual dimensions for less important aspects. However, we should also keep in mind the evolution of visual display technologies, which constrain what is possible at any given time. Only in the 1990s, when people started using computers to design and present visualizations on computer monitors, did color become the norm. Color printing is still significantly more expensive than using a single color—so even today many academic journals are printed in black and white. Thus the extra cost associated with creating and printing color graphics during the last two centuries was likely also a reason for privileging spatial variables in visualization design.
When hue, shading, and other nonspatial visual variables were used in visualizations created in the nineteenth century and most of the twentieth century, they usually represented only a small number of discrete values (typically nominal and ordinal scales). However, computer-based scientific visualization, geo-visualization, and medical imaging today often use such variables to represent continuous values (ordinal and ratio scales). Hue, shading, and transparency are now commonly employed in these fields to show continuously varying qualities such as temperature, gas density, elevation, gravity waves, and so on.
How is this possible? Consider hue. As I already noted earlier, today computers allocate anywhere from eight to forty-eight bytes to represent pixel values in a digital image, so computer displays can show more unique colors than human eyes can see if the monitor can render all these colors. Similarly, displays can show hundreds of levels of shading and transparency, which can also code continuous variables.
Does this contradict my statement that spatial arrangement is key to information visualization? We can solve this puzzle if we consider a fundamental difference between information visualization and scientific visualization/geovisualization, which I did not yet mention. Information visualization uses arbitrary spatial arrangements of elements to represent the relationships between data objects. Scientific, medical, and geovisualization typically show existing (or simulated) physical objects or processes such as a brain, a coastline, a galaxy, an earthquake, and so on. Because the spatial layout in such visualizations is already fixed and cannot be arbitrarily manipulated, color, shading, and other nonspatial variables are used instead to show new information. A typical example of this strategy is a heat map, which uses transparency, hue, and saturation to overlay information over a spatial map.29
The two key principles that I suggested—data reduction and privileging of spatial variables—do not account for all possible visualizations produced during last three hundred years. However, they are sufficient to separate information visualization (at least as it was commonly practiced until now) from other techniques and technologies for visual representation: maps, engraving, drawing, oil painting, photography, film, video, radar, MRI, infrared spectroscopy, and the like. They give information visualization its unique identity—the identity that remained remarkably consistent for almost three hundred years, until the second part of the 1990s.
The meanings of the word visualize include make visible and make a mental image. This implies that until we visualize something, this something does not have a visual form. It becomes an image through a process of visualization.
If we survey the practice of infovis from the eighteenth century until the end of the twentieth century, we indeed see that visualization takes data that is not visual and maps it into a visual domain. However, it seems to no longer adequately describe certain new visualization techniques and projects developed since the middle of the 1990s. Although these techniques and projects are commonly discussed as information visualization, is it possible that they represent something else—a fundamentally new development in the history of representational and epistemological technologies, or at least a new broad visualization method for which we do not yet have an adequate name.
Consider a visualization technique called a tag cloud.30 The technique was popularized by Flickr in 2005 and today it can be found on numerous websites and blogs. A tag cloud shows the most common words in a text, and the size of each word corresponds to its frequency in the text. We can use a bar chart with text labels to represent the same information—which in fact would work better if the word frequencies are very similar. But if the frequencies fall within a larger range, we do not have to map the data into a new visual representation in this way. Instead, we can vary the size of the words themselves to represent their frequencies in the text. This is the idea of the tag cloud technique.
Tag clouds exemplify a broad method that I will call media visualization: creating new visual representations from the actual visual media objects or their parts. Rather than representing text, images, videos, or other media though new visual signs such as points, rectangles, and lines, media visualizations build new representations out of the original media. Images remain images; text remains text. (For examples of media visualizations, see plates 12–16.) In view of our discussion of the data reduction principle, we can also call this method direct visualization or visualization without reduction. In this method, the data is reorganized into a new visual representation that preserves its original form.
Usually, media visualization does involve some data transformation, such as changing data size. For instance, a text cloud visualization would not show every word that occurs in a long text; instead it only shows a smaller number of the most frequently used words. However, this is a reduction that is quantitative rather than qualitative. We do not substitute media objects for new objects (i.e., graphical primitives typically used in infovis), which only communicate selected properties of these objects (e.g., bars of different lengths representing word frequencies). My phrase “visualization without reduction” refers to this preservation of a much richer set of properties of data objects when we create visualizations directly from them.
Not all media visualization techniques, such as a tag cloud, originated in the twenty-first century. If we project this concept retroactively into history, we can find earlier techniques that operate within the media visualization paradigm. For instance, a familiar book index can be understood as a media visualization technique. Looking at a book’s index, you can quickly see if particular concepts or names are important in the book because they will have more occurrences listed; less important concepts will take up less space.
Although both the book index and the tag cloud exemplify the media visualization method, it is important to consider the differences between them. The older book index technique relied on the typesetting technology used for printing books. Because each typeface was only available in a limited number of sizes, the idea that you can precisely map the frequency of a particular word to its font size was counterintuitive—so it was not invented. In contrast, the tag cloud technique is an example of what I call software thinking—the ideas that explore the fundamental capacities of modern software. Tag clouds explore the capacities of software to vary every parameter of a representation and to control it using external data. The data can come from a scientific experiment, from a mathematical simulation, from the body of the viewer in an interactive installation, from sensors, and so on. If we take these software capacities for granted, the idea to arbitrarily change the size of words based on some information—such as their frequency in a text—is something we may expect to be actualized in the process of cultural evolution. In fact, all contemporary interactive visualization techniques rely on the same two fundamental capacities: all parameters of a representation can vary, and their values can be controlled by external data.
The rapid growth in the number and variety of visualization projects, software applications, and web services since the late 1990s was enabled by advances in computer graphics capacities in both hardware (processors, RAM, displays) and software (C, Java, and Python graphics libraries, Flash, Processing, Flex, Prefuse, d3, etc.). These developments popularized information visualization and also fundamentally changed its identity by foregrounding animation, interactivity, and more complex visualizations that represent connections among many more objects than was possible previously.31 But along with these three highly visible trends, the same advances also made possible the media visualization approach.
In this section, I will discuss three well-known digital projects that for me exemplify the media visualization paradigm. These projects are Cinema Redux, Preservation of Favored Traces, and Listening Post.32
Cinema Redux was created by interactive designer Brendan Dawes in 2004.33 Dawes wrote a program in Processing that sampled a feature film at the rate of one frame per second and scaled each frame to 8 × 6 pixels. The program then arranged these sampled frames in a rectangular grid, with every row representing a single minute of the film. Although Dawes could have easily continued this process of sampling and remapping—for instance, representing each frame though its dominant color—he chose instead to use the actual, scaled-down frames from the film. The resulting visualization represents a trade-off between the two possible extremes: preserving all the details of the original artifact and abstracting its structure completely. A higher degree of abstraction may make the patterns in cinematography and narrative more visible, but it would also remove the viewer further from the experience of the film. Staying closer to the original artifact preserves the original detail and aesthetic experience but may not be able to reveal some of the patterns.
Ultimately, what is important in the context of our discussion is not the particular sampling values Dawes used for Cinema Redux but that he reinterpreted the previous constant of visualization practice as a variable. Throughout visualization history, infovis creators mapped data into new diagrammatic representations consisting of graphical primitives. This was the default practice. With computers, a designer can now select any value on the original data/abstract representation dimension. In other words, a designer can now choose to use graphical primitives or the original images exactly as they are—or any format in between. Thus, though the project’s title, Cinema Redux, refers to the idea of reduction, in the historical content of earlier infovis practice it can be actually understood as expansion: expanding typical graphical primitives (points, rectangles, etc.) into actual data objects (film frames).
Before software, creating a visualization typically involved a two-stage process: first counting or quantifying data and then representing the results graphically. Software allows for direct manipulation of the media objects without quantifying them. As demonstrated by Cinema Redux, these manipulations can successfully make visible the relations among a large number of these artifacts (thousands of film frames). Of course, such visualization without quantification is made possible by the a priori quantification required to turn any analog data into a digital representation. In other words, it is the reduction first performed by the digitization process that now allows us to visualize the patterns across sets of analog artifacts without reducing them to graphical signs.
For another example of media visualization, let’s look at Ben Fry’s Preservation of Favored Traces (2009).34 This project presents an interactive animation of the complete text of Darwin’s On the Origin of Species. Fry used different colors to show the changes made by Darwin in each of six editions (1859–1872) of his famous book. As the animation plays, we see the evolution of the book text from edition to edition, with sentences and passages deleted, inserted, and rewritten. In contrast to many animated visualizations that show some spatial structure constantly changing its shape and size in time, reflecting changes in the data (e.g., the changing structure of a social network over time), in Fry’s project the rectangular frame containing the complete text of Darwin’s book always stays the same; what changes is its content. This allows us to see how over time the pattern of the book’s additions and revisions becomes more and more intricate as the changes from all the editions accumulate.
At any moment in the animation we have access to the complete text of Darwin’s book—as opposed to only a diagrammatic representation of the changes. At the same time, it can be argued that that Preservation of Favored Traces does involve some data reduction. Given the typical resolution of computer monitors and web bandwidth, Fry was not able to show all the actual book text at the same time.35 Instead sentences are rendered as tiny rectangles in different colors. But when you mouse over any part of the image, a pop-up window shows the actual text. Because all the text of Darwin’s book is easily accessible to the user in this way, I think that this project can be considered an example of media visualization.
Let’s add one more example—Listening Post by Ben Rubin and Mark Hansen (2001).36 Usually this work is considered to be the finest example of the genre of computer-driven installations rather than an information visualization, but I think it is appropriate to also consider it as the latter, expanding classical infovis in a new direction (media visualization). Listening Post pulls text fragments from online chat rooms in real time, based on various parameters set by the authors, and shows them across a display wall made from a few hundred small screens in a six-act looping sequence. The work consists of six “acts” that follow each other. Each act uses its own distinct spatial layout to arrange dynamically changing text fragments. For instance, in one act the phrases move across the wall in a wave-like pattern; in another act words appear and disappear in a checkerboard pattern. Each act also has a distinct sound environment driven by the parameters extracted from the same text that is animated across the display wall.
We can argue that Listening Post is not a visualization because the spatial layouts in all acts are prearranged by the authors and not driven by the data. We can contrast this with classical infovis methods such as a scatter plot, in which the layout of points is derived from the data. This argument does make sense—but it is important to keep in mind that though layouts are prearranged, the data in these layouts is not; it is a result of the real-time data mining of the web. So although the text fragments are displayed in predefined layouts (wave, checkerboard, etc.), because the content of these fragments is always different, the overall result is also always unique. In contrast, in a scatter plot all points are often exactly the same, so information is only carried by their layout.
Note also that if the authors were to represent the text via abstract graphical elements, we would simply end up with the same abstract pattern in every repetition of an act. But because they show the actual text that changes all the time, the patterns that emerge inside the same layout are always different. This is why I consider Listening Post to be a perfect representative of media visualization thinking: the patterns it presents depend as much on what all the text fragments that appear on the screen wall actually say as on their predefined composition.
Whereas common visualization techniques such as scatter plots and bar charts only define a basic mapping method and the rest is driven by the data (positions of points, length of bars), visualizations of networks use a different principle. Like Listening Post, they use certain predefined layouts. Manuel Lima identified what he calls a syntax of network visualizations—commonly used layouts such as radial convergence, arc diagrams, radial centralized networks, and others.37 The key difference between many of these network visualizations and Listening Post lies in the fact that the former often rely on the existing visualization layout algorithms. Thus, they implicitly accept the ideologies behind these layouts—in particular the tendency to represent a network as a highly symmetrical and/or circular structure. The authors of Listening Post wrote their own layout algorithms that allowed them to control the layouts’ intended meanings. It is also important that they used six very different layouts that cycle over time. The meaning and aesthetic experience of this work—showing both the infinite diversity of web conversations and the existence of many repeating patterns—to a significant extent derive from the temporal contrasts between these layouts. Eight years before Bruno Latour’s article (quoted at the beginning of the chapter) in which he argues that our ability to create “a provisional visualization which can be modified and reversed”38 allows us to think differently because any “whole” we can construct now is just one among numerous others, Listening Post beautifully staged this new epistemological paradigm, enabled by interactive visualization.
The three visualizatio projects I considered demonstrate that in order to highlight patterns in the data we do not have to dramatically reduce it by representing data objects as abstract graphical elements. We also do not have to summarize the data as is common in statistics and statistical graphics; think, for instance, of a histogram that divides data into a number of bins or a bar chart showing the numbers of items in multiple categories. This does not mean that in order to qualify as a media visualization an image has to show 100 percent of the original data—every word in a text, every frame in a movie, or so on. Out of the three examples I just discussed, only Preservation of Favored Traces does this. Both Cinema Redux and Listening Post do not use all the available data; instead they sample it. The first project samples a feature film at the fixed rate of one frame per second; the second project filters the online conversations using set criteria specific to each act. However, what is crucial is that the elements of these visualizations are not the result of remapping of the data into some new representation format; they are the original data objects selected from the complete dataset. Images remain images, text remains text. This strategy can be related to the traditional rhetorical figure of synecdoche—and specifically the particular case in which a specific class of thing refers to a larger, more general class.39 (For example, in Cinema Redux, one frame stands for a second of a film.)
Although sampling is a powerful technique for revealing patterns in the data, Preservation of Favored Traces demonstrates that it is possible to reveal patterns while keeping 100 percent of the data. But you have likely employed this strategy—for instance, if you have ever used a magic marker to highlight important passages of a printed text (or the equivalent of this technique in a word processor). Although text highlighting normally is not thought of as visualization, we can see it as an example of a media visualization that does not rely on sampling. You sample selected sentences from the complete texts and highlight them so that later you can only look at these sentences.
Preservation of Favored Traces and Cinema Redux also both break away from the second key principle of traditional visualization: communication of meaning via spatial arrangement of elements. In both projects, the layout of elements is dictated by the original order of the data—shots in a film, sentences in a book. This is possible and also appropriate because the data they visualize is not the same as the typical data used in infovis. A film or a book is not just a collection of data objects: it is a narrative made from these objects. Of course, infovis designers also often work with sequential data (e.g., time measurements in an experiment, quarterly sales for some products, volumes of social media posts over time). But these are not a priori narratives. And usually we visualize only a single dimension of such data: quantities over time. But the book or a movie is a truly “thick” narrative in which changes take place over multiple dimensions, and events and scenes are connected to others appearing much earlier and later.
It is certainly possible to create effective visualizations that remap a narrative sequence into a completely new spatial structure, as in Listening Post (see also Writing without Words by Stefanie Posavec and The Shape of Song by Martin Wattenberg40). But Cinema Redux and Preservation of Favored Traces demonstrate that preserving the original sequences is also effective.
Preserving the original order of data is particularly appropriate in the case of cultural datasets that have a time dimension. I call such datasets cultural time series. Whether feature films (Cinema Redux), books (Preservation of Favored Traces), or long Wikipedia articles (History Flow), the relationships between the individual elements (a film’s shots, a book’s sentences) and between larger parts of a work (a film’s scenes, a book’s paragraphs and chapters) separated in time are of primary importance to the work’s evolution, meaning, and user experience. We consciously or unconsciously notice many of these patterns while watching, reading, or interacting with the work, but projecting time into space—laying out movie frames, book sentences, magazine pages in a single image—gives us new possibilities to study them. Thus, space turns out to play a crucial role in media visualization after all: it allows us to see patterns between media elements that are normally separated by time.
In an article on the then-emerging practice of artistic visualization written in 2002, I defined visualization as “a transformation of quantified data which is not visual into a visual representation.”41 At that time, I wanted to stress that visualization participates in the reduction projects of modern science and modern art, which led to the choice of the article’s title: “Data Visualization as New Abstraction and Anti-Sublime.” I think that this emphasis was appropriate given the types of infovis projects typically created at that time. (Although I used a somewhat different formulation for the definition that appears at the beginning of this chapter—a remapping from other codes to a visual code—the two definitions focus on the same concept: mapping.)
Most information visualization today continues to employ graphical primitives. However, as the examples in this chapter demonstrate, alongside this “mainstream” infovis, we can find another trend. These are the projects in which the data being visualized is already visual—text, film frames, magazine covers. In other words, these projects create new visual representations from the original visual data without translating it into graphic signs. They also show that the second key principle of normal infovis—mapping of the most important data dimensions into spatial variables—does not always have to be followed.
Does media visualization actually constitute a form of infovis, or is it a different paradigm altogether? We have two choices. Either we accept that this is something fundamentally different, or, alternatively, we can revise our understanding of what infovis is. Given that all media visualizations we looked at aim to make visible patterns and relations in the data, this certainly aligns media visualization with infovis as it developed during the last three hundred years. It is also relevant that some of the most well-known infovis projects of the last twenty years follow a media visualization approach. This is true of Cinema Redux and Preservation of Favored Traces and other seminal projects that I did not discuss, such as Talmud Project (David Small, 1999),42 Valence (Ben Fry, 2001),43 and TextArc (W. Bradford Paley, 2002).44 This means that people intuitively identify them as infovis projects even though they consist not of vector elements but of media (text or images). In another example, a phrase net technique, which was developed by Frank van Ham, Martin Wattenberg, and Fernanda Viégas and awarded Best Paper at the IEEE InfoVis 2009 conference, also uses a media visualization paradigm.45
Does this mean that what we took to be the core principle of information visualization during its first three centuries—reduction to graphic primitives—was only a particular historical phenomenon, an artifact of the available graphics technologies? I think so. Similarly, the privileging of spatial variables over other visual parameters may also turn out to be a historically specific strategy rather than the essential principle of infovis. The new abilities brought about by computer graphics to precisely control—that is, assign values within a large range—color, transparency, texture, and many other visual parameters in any part of an image allow us to start using these nonspatial parameters to represent the key dimensions of data. This is already common in scientific, medical, and geovisualization—but not yet in information visualization.
Why did infovis designers continue to use computer-generated vector graphics during the 1990s and 2000s (i.e., using the visual language developed 170 to 200 years ago, but only now with computers) while the speed with which computers could render images was progressively increasing? Perhaps the main factor has been the focus on the World Wide Web as the preferred platform for delivering interactive visualizations. Web technologies made it relatively easy to create vector graphics and stream videos—but not to render large numbers of continuous tone images in real time. This required the use of a graphics workstation, a high-end PC with a special graphics card or a game console with optimized graphics processors, as well as time-consuming software development. Although video games and 3-D animation programs could render impressive numbers of pixels in real time, this was achieved by writing code that directly accesses hardware—something that very high-level media programming environments such as Processing and Flash/Flex could not do.
However, as the processing power and RAM size of personal computers and computer devices (desktops, laptops, tablets, phones, etc.) keeps increasing, these differences between the performance of programs written in high-level languages and the programs that work on lower machine levels become less important.
For example, in 2009 I developed ImagePlot visualization software for our work in the lab.46 For a programming language, I used the high-level scripting language of ImageJ, a popular open-source application for image processing used in the sciences.47 Running ImagePlot on my 2010 Apple PowerBook laptop (processor: 2.8 GHz Intel Core 2 Duo; memory: 4 GB), I rendered a media visualization showing the evolution of design and content in 4,535 Time magazine issues (1923–2009). The visualization resolution was 30,000 × 4,000 pixels, and it took only a few minutes to render it. Most of that time was spent scaling down the images from their original size to the small size used in the visualization (see plate 12).
Also, in 2009 we developed the HiperView48 software together with Calit2’s Center of Graphics, Visualization and Virtual Reality (GRAVITY). The software was created to run on visual supercomputers constructed by this lab. Its largest system at that time was a tiled display with a 286-megapixel resolution made from seventy 30-inch Apple monitors all running at 2,560 × 1,600 resolution and a number of PCs with high-end graphics cards. The software enabled a researcher to manipulate media visualizations interactively in real time showing up to ten thousand images of any size. For example, you could position all the Time covers by time (x-axis) and then instantly see temporal patterns on many dimensions by choosing this or that visual feature or metadata for the y-axis (gender and ethnicity of a person on a cover, average saturation and hue, etc.). You could also load a very high-resolution visualization of one million manga pages and instantly zoom in, pan to explore its details, and zoom out to see their context (see plate 10).
I believe that the media visualization approach is particularly important for humanities, media studies, and cultural institutions. Many of these only recently have started to discover the use of visualization but eventually may adopt it as a basic tool for research, teaching, and exhibition of cultural artifacts. (The first conference on visualization in the humanities took place at MIT in 2010.49)
If all media and humanities scholars start systematically using visualization for research, teaching, and public presentation of cultural artifacts and processes, the ability to visualize media artifacts in full detail is crucial. Displaying the actual visual media in the dataset as opposed to representing it by graphical primitives helps the researcher to understand the meaning and/or cause behind the pattern she may observe, as well as to discover additional patterns.
Graphical reduction will continue to be used, but this is no longer the only possible method. The development of computers and the progress in their media capacities now makes possible a new visualization paradigm that I have called media visualization—visualization that does not reduce original media artifacts to points, bars, or lines. Instead, it displays all the artifacts in the dataset in their original form, and sorts, samples, and remaps them in many ways to make possible new discoveries.50