What does it mean to represent a cultural object, process, or experience as data that can be then analyzed computationally? What elements of these objects, processes, and experiences can be captured, and what remains outside? How can we represent people interactions with computational cultural artifacts and systems that can react to human behaviors, communicate with them (e.g., as AI interfaces can), and act in seemingly intelligent ways? These are all fundamental questions for cultural analytics.
In this chapter, I will look at four categories of “things” in global digital culture that we can analyze computationally on a large scale. Note that “digital” here refers to both the phenomena that are native to computer devices and networks (making posts, sharing media, commenting, participating in online groups and forums, using apps) and the physical phenomena that are represented in digital universe (e.g., webpages for organizations and events). The four categories we will examine are media, behaviors, interactions, and events.
Media here refers to digital artifacts created by professional creatives and users of social networks. Behaviors include both online activities that leave digital traces and physical behaviors that can be captured using other methods. Interactions refers to the activity of using interactive, algorithm-driven media such as video games, virtual earth software such as Google Earth, or virtual reality and augmented reality applications. Finally, events are cultural happenings that have duration in time and involve multiple people: a music performance, an exhibition opening, a fashion show, a workshop, a weekend urban festival, a demonstration by a master coffee maker. These events usually take place in specific places and are presented by organizations, so this category also includes these entities.
These four categories are not meant to include every phenomenon in digital culture. As an example of other equally important phenomena, consider networks. There is an academic field called network science devoted to the study of complex networks and development of theories and ways to measure networks. The network paradigm—that is, seeing a phenomenon as a network—also has been central to other fields since the 1990s. This paradigm assumes that the structure and characteristics of the networks are more important than any individual nodes and links. Adopting this paradigm for the study of digital culture means focusing on a network and the movement of media objects, topics, people, and other objects and actions within it—away from singular media artifacts, behaviors, or events. For example, the authors of the 2018 paper “Quantifying Reputation and Success in Art” performed network analysis of the art world. The authors analyzed data on “497,796 exhibitions in 16,002 galleries, 289,677 exhibitions in 7568 museums, and 127,208 auctions in 1239 auction houses, spanning 143 countries and 36 years (1980 to 2016).”1 The analysis reveals important patterns in how artists move over time through the network of all these galleries and museums; for example, the artists that had their first five exhibitions in high-prestige institutions were likely to continue exhibiting in such institutions. In contrast, among artists who had their first five exhibitions in institutions ranked in the bottom 40 percent, only 14 percent continued to be active ten years later. The initial exhibition history of an artist was also predictive of other measures of success: “High–initial reputation artists had twice as many exhibitions as low–initial reputation artists; 49% of the exhibitions of high–initial reputation artists occurred outside of their home country, compared to 37% for low–initial reputation artists, and high–initial reputation artists showed more stability in institutional prestige.” These and other patterns were revealed by considering the art world as a big network, with artists and artworks moving in it—without having to take into account the styles and content of the artworks themselves or biographic details of the artists.
There are also other ways to look at the digital universe in terms of things that we can select for analysis. For example, we can distinguish among cultural data, cultural information, and cultural discourse:
Another important distinction is between the original cultural artifact/activity and its digital representations:
Last but not least, we can also approach digital universe through the lenses of three categories that have been standard in humanities and media studies: authors, texts (or messages), and audiences. We can collect and analyze large data on authors, messages, and audiences using many methods—for example, network analysis of the connections between a group of authors, computer vision analysis of content of visual media they created, spatial analysis of people movements in a museum in relation to the exhibits, and so on. (Texts or messages in this scheme correspond to “media” as used in this chapter.)
Since the middle of the 2000s, large-scale global social and media sharing networks and messaging services such as Facebook, Twitter, Baidu, VK, Flickr, Instagram, Tumblr, Snapchat, WhatsApp, WeChat, Weibo, and LINE have aggregated massive amounts of posts, images, videos, comments, and discussions contributed by billions of people. User-generated content and interactions on some of these networks have been extensively analyzed by researchers in computer science, computational social science, and other fields.
However, it is crucial to remember that “social media” is not limited to content shared on these networks. The following list of types of social media with examples comes from the call for papers for the 11th Annual International Conference on Web and Social Media (2017):
- • Social networking sites (e.g., Facebook, LinkedIn)
- • Microblogs (e.g., Twitter, Tumblr)
- • Wiki-based knowledge sharing sites (e.g., Wikipedia)
- • Social news sites and websites of news media (e.g., Huffington Post)
- • Forums, mailing lists, newsgroups
- • Community media sites (e.g., YouTube, Flickr, Instagram)
- • Social Q & A sites (e.g., Quora, Yahoo Answers)
- • User reviews (e.g., Yelp, Amazon.com)
- • Social curation sites (e.g., Reddit, Pinterest)
- • Location-based social networks (e.g., Foursquare, Pokémon Go)
- • Online dating sites (e.g., Match, Tinder, Grindr)
- • Messaging platforms (e.g., Snapchat, Messenger, WhatsApp)2
The rise of online marketplaces such as Amazon, social media networks, and blogs in the 2000s created a new environment in which people voluntarily reveal their cultural choices and preferences: rating books, movies, software, images, videos, and songs; assigning likes and favorites; sharing cultural posts by others; and more. People explain, defend, and debate their cultural preferences, ideas, and perceptions. They comment on Instagram photographs, post their opinions about books on Amazon, critique movies on Rotten Tomatoes (rottentomatoes.com), and enthusiastically debate, argue, and agree and disagree with each other on numerous social media sites, fan sites, forums, groups, and mailing lists.
APIs, web scraping, and social media monitoring software give us access to large samples of content and user activities on these sites and networks. Conversations, discussions, and opinions can be analyzed computationally to find out what cultural topics are important to people in dozens of countries and how they see culture.
We can also find out which characteristics of cultural forms and genres people discuss in online forums. These characteristics then can be compared to the vocabularies used by professionals who create these forms and to the languages of academic researchers who write about them. (We are likely to find that fans pay attention to different things than casual viewers and academics.)
One social media type that does not appear in the earlier list but is particularly important for cultural analytics is networks and media sharing sites for culture professionals, aspiring creators, and students in creative fields. I described early examples of such sites, such as Coroflot, in chapter 1. In the first part of the 2000s, social networks were mass phenomena only in Japan and Korea, not yet anywhere else. At the time we set up our lab, Flickr was the only visible media sharing site in the United States, having reached two million users by October 2007. And though it was already possible to add images and videos to blogs, the process was not very simple, and not many blogs had visual media.3 But portfolio sites such as Coroflot for professionals and students in creative fields were already fully operational.
Today, prominent examples of English-language professional portfolio and project sites include Behance (2006–) and dribble (2009–) for all types of design and visual communication, Archinect for architecture (1997–), 500px for photography (2009–), and DeviantArt for user-generated art (2000–). For data scientists, the equivalent of online portfolios is participation in competitions on kaggle.com. For professional writers and journalists, there are Clippings.me, Muck Rack (muckrack.com), and others. For music creators, there is SoundCloud (2007–). Video, motion graphics, and animation professionals use Vimeo (2004–). And all members of the “creative class” can also advertise themselves and look for jobs on freelance sites such as Upwork, Guru, Freelancer, and many others. There are also many similar sites in other languages, and many such sites have APIs.
Professional networks such as Behance are particularly good sources of data for analyzing global professional culture because their users share important information in addition to the examples of work projects: CVs, descriptions of projects, lists of clients, and personal statements. Housing projects and portfolios by millions of artists, media designers, freelance writers, coders, and other culture professionals, such websites provide live snapshots of contemporary global cultural production, sensibilities, and emerging areas. They also make possible analyzing changes in content, styles, techniques, and use of various media over time.
One of the earliest media sharing networks aimed at aspiring creatives rather than everybody was DeviantArt (deviantart.com; 2000–). By August 2008, it had eight million members, more than sixty-two million submissions, and was receiving eighty thousand submissions per day.4 By 2016, DeviantArt had thirty-eight million registered users who were uploading 160,000 original artworks every day.5 In our lab, we investigated changes in subjects, techniques, sizes, proportions, and selected visual characteristics of images in one million visual artworks shared on DeviantArt during the ten-year period from 2001 to 2010 (see chapter 7 for more details about this project).
Another interesting source for professional content is specialized groups on media sharing sites. For example, the group for motion graphics on Vimeo called Motion Graphics Artists had approximately 110,000 videos shared by thirty thousand members in March 2017.6 Vimeo also has groups created by artists from various countries and cities, such as Look at Russia, @Mograph Spain, and Motion Graphics New York. We can use works shared by members of these groups to compare design trends between different countries. In one study, we have compared 170,000 images from the Graphic Design Flickr group with the same number of images from the Art Now Flickr group. As our paper explains, one of the purposes of this analysis was to learn “how services such as Flickr influences artists to pool their work together. We speculate that artists often decide to join a particular group by browsing its content and doing the mental computation to understand patterns in this content, the process that we aim at imitating with our algorithms.”7
In addition to the obligatory presence on general social networks and professional networks such as Behance, today a serious creative professional maintains their dedicated site with a unique URL. A dedicated website is also the default medium to present a large or ongoing project or event in most cultural fields, in addition to a page on Facebook. How many such websites for cultural professionals, projects, and organizations are created or updated every year? As far as I know, nobody has ever bothered to ask such question. And this is a telling example of how little we know about contemporary global culture. The rapid expansion of the digital cultural universe is like an expansion of our physical universe after a big bang a billion years ago (according to the dominant cosmological paradigm). Manually browsing the web or using recommendations provided by social networks and sharing sites, we can see only tiniest corners of the continuously expanding cultural universe. But if we used proper sampling methods to collect big datasets and analyze them with data science methods, what we would be able to see would no longer be limited by lists of automatically generated “trending topics” or inaccessible algorithms underlying recommendation systems or lists of “top” or “most important” items created by some experts and geared to the interests of certain audiences. Such data collection methods include using APIs when they are available and downloading data via web crawling or web scraping when APIs do not exist. Using the latter methods, we can access and then analyze all kinds of cultural content outside social networks. (Of course, we also need to follow accepted guidelines to protect user privacy when using data from websites, social media networks and other online sources. I will discuss this later in this chapter.)
This data potentially can be used to do something that previously was unimaginable: to create dynamic, interactive, detailed maps of global cultural developments that reflect the activities, aspirations, and visions of millions of creators. Such dynamic maps will have a sufficient temporal and geographic resolution to show how trends emerge, travel in space and time, change in popularity, are combined with other trends, and so on (see plate 2).
But we also should not underestimate the challenges. For instance, though digitization and networking of culture gave us a massive volume of artifacts, it also led to the proliferation of new media forms and genres. And if humanities and media studies did not keep up with this proliferation, neither did computer sciences. Here older media popular by the end of the twentieth century—photographs, videos, songs, spatial data and text (including websites, blogs, and social media posts)—received the most attention. Automatic recognition and detection of objects, types of scenes, and higher-level concepts in photos has been a key focus of research in computer vision, and significant progress was made in the 2010s. Good progress has also been achieved for particular types of content, such as fashion photography. Significant accuracy has been achieved in automatic analysis of professional and nonprofessional fashion photos, including detecting types of clothes, poses, brands, and styles (hipster, bohemian, preppy, etc.).8 Another research area where we see promising results is detection of illustration styles.9 But at least so far, automatic analysis of styles, techniques, and forms in graphics, products, interfaces, and UI of games has not been posed as a research problem; the first paper on automatic classification of styles in architecture was only published in 2019.10 (The adoption of neural networks that perform well for classification of new types of cultural data is a good start, but we certainly don’t have anything yet that is similar in depth and breadth to the methods in natural language processing for other media.)
In fact, the global use of networked computers and devices for creating and sharing content and communication led not only to the quantitative explosion of cultural artifacts since the middle of the 2000s, but also to growth in number of media genres and their combinations: HDR photos, photos with filters, photos with superimposed stickers and drawings, 360-degree video, vertical video, virtual globes, emoji, screen icons, adaptive web designs, and more.11 These genres, types, and combinations often change and evolve quickly. This is possible because they all “live” within a strictly software environment, so on a material level nothing needs to change; everything continues to be made from the same pixels, vectors, and text characters.12 These dynamics and this mutability present a significant challenge for any type of media analysis today, be it nonquantitative media theory or quantitative studies that rely on algorithmic analysis of large samples. In other words, as algorithms to analyze some genres achieve better performance and researchers finally start paying attention to new forms, new genres and new forms keep emerging, and the older ones keep changing.
It is possible to argue that cultural things in our first category—user-generated and professionally created digital artifacts—are not different from historical artifacts we have from the past, despite their much greater volume, variety, and velocity. Both today and in the past, we have cultural objects created by individuals or groups: builders of a temple, cathedral, or a church eight hundred years ago, a movie studio fifty years ago, or a game design or software company today.
Similarly, you can argue that although reviews are now being created on a massive scale (e.g., Yelp, TripAdvisor, IMDB), they also existed in the past, written by professionals or educated elites. The same can be said about millions of creatives’ CVs now shared on job-hunting and portfolio sites. We also have biographies for some professional artists, writers, scientists, and other members of professional and knowledge classes and other creatives who worked in earlier centuries. An article by Maximilian Schich and his coauthors titled “A Network Framework of Cultural History” demonstrates how a new vision of cultural history can emerge by following locations of birth and death of 150,000 notable individuals over two thousand years.13
What we certainly don’t have from the past are detailed and large-scale automatic recordings of cultural behaviors in large numbers: discussing, reading, listening, viewing, gaming, navigating, searching, exploring, collaborating. But once these activities involve computers, the situation changes. Quantitative large-scale studies of what cultural theory refers to as reception become possible. Any activity that passes through a computer or a computer-based media device—surfing the web, playing a game, participating in a chat, sharing a post, writing a comment, editing a photo—automatically leaves traces: keystroke presses, cursor movements, controller positions, selected items on a menu, commands typed, and so on. The Undo command and History window present in many applications illustrate this well. Web servers log which pages users visited, how much time they spent on each page, which links they clicked on, and so on. Apps developers use special software services (e.g., Google’s Firebase) to record and analyze detailed actions taken by users of their apps.
Thus, it is our second category of cultural things—behaviors—that turns out to be most different in terms of data available for cultural research. Between 2007 and 2018, an unprecedented amount of data on online behaviors was available via social network APIs. (As of 2018, US-based networks have limited third-party access to their data to comply with the EU General Data Protection Regulation and also in response to the news that their data was used to manipulate voters’ opinions in the 2016 US presidential election and the 2016 Brexit vote in the United Kingdom.) In some of these years, APIs from some of the most popular networks allowed access to detailed information for all public posts, including geographic location, the date and time when the post was made, numbers of likes, username of a person who made the post, its text, hashtags, shared images or videos if present, and more. The APIs also made available many types of data about each user. For example, in 2017 the Twitter API provided over forty different pieces of such data, such as self-declared account location and interface language, image used for the profile, IDs and counts of friends, and others.14 The availability of all this data played a key role in the development of computational social science in the second part of the 2000s.
If we use social media data accessible via APIs for research purposes, we have to follow rules carefully to ensure user privacy. This was not obvious at first, but eventually researchers in computer science and computational social science started to do this, and in 2013 national bodies such as the US Department of Health and Human Services created detailed guidelines.15 Companies such as Facebook also created guidelines and review boards for data use by their own researchers after the topic of access to user data started to be widely discussed.16 Research and publication guidelines by national bodies and academic journals state that any personal data can be used only with explicit user consent. According to one such set of guidelines, in situations where “it is not possible to guarantee that personal data will not be collected,” the researchers should delete such data after collection.17 I have already mentioned industry projects such as Facebook’s Social Science One that aim to provide big data to researchers without comprising privacy, but this is only one possible solution. As digital culture researchers Tommaso Venturini and Richard Rogers pointed out in 2019: “The closure of APIs . . . can have positive effects, if it encourages researchers to reduce their dependence on mainstream platforms and explore new sources and ways to collect online records that are closer to the digital fieldwork.”18
Of course, not all cultural activities and behaviors are mediated by computers. People attend concerts and go museums, read books, spend time in their favorite cafes, travel, and go clubbing. When friends meet, they may browse a magazine together, discuss fashion or food trends, and show each other photos on their mobile photos and comment on them. These are physical activities, even though they probably first used the web to find information about these places and to make reservations to eat together and took photos with their camera phones.
During the twentieth century, social scientists developed various methods for qualitative research, including participant observation; field notes; structured, semi-structured, and unstructured interviews; case studies; thick description; and a number of others. These methods are certainly appropriate for the study of cultural behaviors—especially if they are taking place in a physical space like in my earlier examples. Observation, interviews, and participation in activities allow a researcher to understand the motivations of actors and the meanings of an activity.
Interestingly, though recently humanities scholars have shown a strong interest in quantitative approaches, the universe of qualitative methods and the nuanced theoretical discussions about them in the social sciences remain practically unknown in humanities. The reasons for this are likely to do with the historical orientation of the humanities; you can’t interview nineteenth-century novel readers or run a focus group with the public that attended the first movie screenings in 1895. But if we are concerned with contemporary culture, qualitative methods should be among our tools. (See work in the field of digital ethnography, which uses qualitative methods for study of online communities.)
Although directly observing or participating in a cultural activity, doing interviews, or embedding yourself in a group for a significant period of time are powerful methods, we may not observe everything. Capturing records of human behavior with technology is often helpful—not only because of the possible scale of such observations but also because we can capture dimensions of experiences that participants may not be able to report verbally or evaluate correctly. Combined with a video camera, microphone, GPS device, and other sensors, computer devices can capture many aspects of human physical behaviors and physiological states, such as speech, eye movements, geographic locations, positions of body parts, pulse, electrical and blood and brain activity (using EEG and fMRI), and others.
Such recordings are widely used in the culture industry and IT industry. For example, eye movement recordings are employed in testing advertising and for evaluation of computer and product interfaces and website designs. The production of numerous video games and movies relies on motion capture, in which movements of actors’ bodies and faces are recorded and then used to animate computer-generated characters.
Capturing aspects of participants’ and performers’ behaviors has been also an important strategy in interactive art, dance, performance, and music. Artists, dancers, and musicians have been using video capture and sensors to capture body positions and movements for many decades. The prominent early examples include a series of interactive works created by Myron Krueger since 1969, such as Videoplace, as well as Very Nervous System by David Rockeby (1981–). Artists Joachim Sauter and Dirk Lüsebrink used eye movement capture in their work Zerseher (1992) in a very simple but conceptually powerful way. A viewer was presented with a monitor showing a painting. Using eye tracking, the viewer’s gaze was captured: “As the viewer looks at the painting, the painting begins to change in the exact places where the gaze of the viewer is pointed.”19
One of the pioneers of eye movement research was Russian psychologist Alfred Yarbus. In his influential book Eye Movements and Vision (1962), translated into English in 1967, he analyzed a series of recordings of eye movements of human subjects viewing a well-known nineteenth-century realistic painting by Ilya Repin. This classical study anticipated the wide use of eye movement recordings in advertising and design, and it has been replicated many times by other researchers in different countries.20 Yarbus used this painting to show that a task given to a person dramatically affects their eye movements. He wrote: “Eye movement reflects the human thought processes; so the observer’s thought may be followed to some extent from records of eye movement (the thought accompanying the examination of the particular object). It is easy to determine from these records which elements attract the observer’s eye (and, consequently, his thought), in what order, and how often.”21
Today, people’s geographic locations and body movements are captured on a large scale by mobile phones, fitness trackers, and other wearables. This is, for example, how Google creates the graphs for many businesses that show their popularity for every hour and day of the week: it collects location data captured from phones of users who agreed to share that data. Fitness apps and trackers capture data about types of exercise, speed and intensity, and the duration and density of sleep. And don’t forget the many millions of video cameras in our cities and taxi cars, road sensors, GPS devices on bicycles: a giant portrait of humanity engaged in walking, sleeping, driving, bicycling, sitting, lifting, running—a portrait of the human species on Earth as physiological organisms, distributed between multiple systems, file formats, servers, locations, and organizations.
Some parts of this mega portrait have been accessible to researchers. Aggregated anonymized location data has been used in thousands of quantitative studies of tourists’ and locals’ mobility patterns around the world and inside cities. These studies appear in academic journals in geography, urban studies, tourism studies, environmental science, transportation, and new fields that developed in the second part of the 2000s at the intersection of urban research and computation—urban informatics, urban computing, and the science of cities. For example, in a 2015 study, researchers from the Sensible City Lab at MIT combined location data from 3.5 million Flickr photos, twenty-four million tweets, and anonymized bank transactions from three hundred thousand ATMs in Spain to investigate the relationship between visitor activity and city size.22 In our work, we have been collaborating with two urban data analysis groups—Habidatum and SPIN Unit. One of SPIN Unit’s projects was an analysis of seventy monotowns in Russia using locations and time stamps from millions of posts on VK, the largest social network in Russia. (A monotown is a city in which most people work or used to work in one big enterprise. In many such cities in Russia, these enterprises have closed, and how to sustain and improve people’s lives in these cities is a big issue.) SPIN Unit determined which areas of cities have more people making posts and at what time, and also made some predictions of types of activities. This project was commissioned by the leading Russian urban design agency Strelka KB to develop guidelines for improving life in these cities via urban interventions.
Sensible City Lab, SPIN Unit, Habidatum, and other groups and labs are frequently commissioned by city agencies around the world to run similar projects, using data from many sources, including vehicles.23 For example, the New York City bike-sharing program publishes data about bike rides, and its website has a section featuring many artistic visualizations and other creative projects created with this data. For a good example that shows what we can learn from data released by the city bike share system, see “A Tale of Twenty-Two Million Citi Bike Rides.”24 In the future, we are likely also to see visualizations and installations that use data generated by self-driving cars; each car is expected to generate 4 TB of data per day.25 And of course there will also be data created by all the future robots.
As more and more cities around the world implement “smart city” strategies, massive collections of data about people, vehicles, and devices and their behaviors become more commonplace. This brings out many political and social issues—privacy, access to collected data (such as whether this data can be used by citizens or only by city agencies or private companies), and whether efficiency and economy of resources should be the main goal of a smart city. So far, the old modernist efficiency goal has dominated smart city discussions, at the expense of other potential uses of urban data such as increasing diversity and variability of urban planning and design, and supporting more spontaneous behaviors.
Today location and movement data are utilized by urban planners, engineers, city agencies, and policymakers, while companies such as Uber and Waze share some of their car trip data with researchers.26 But though such data may look massive and fine-grained today, it is only the beginning. In the future, records of human physiological states and the brain activity of hundreds of millions of people may also become widely available—and widely used. Will millions of people living in megacities agree to make their real-time brain activity and eye movement recordings available to urban planners and architects so they can improve cities? Or will such data collection be required for citizens to have the rights to use basic city services? Or . . . add your own scenario.
To conclude this section, I want to highlight my main point. Analyzing culture at scale computationally means more than only using large collections of media artifacts or records of users’ digital behaviors in social networks. The concept of culture also includes physical behaviors, experiences, moods, feelings, and emotional states. Depending on our goals, we may want to analyze these other dimensions. For example, in the case of spatial experiences, architectural plans and photos will only tell us a part of a story; we may also want to use video and motion capture of people interacting with the spaces. Or we may simply spend time and observe what people do, using a notebook and a camera as our only tools.
The systematic observation and capture of people’s behavior and interactions for further analysis is nothing new for ethnography, anthropology, urban studies, sport, medicine, and other fields. For instance, French scientist Étienne-Jules Marey invented many devices for capturing movements of humans and animals starting in the 1860s; in the 1910s, Lillian and Frank Gilbreth started to use film recording for motion studies of work processes. In 1969, urbanist William H. Whyte started to use notebooks and cameras to observe people behaviors in New York City streets and public spaces, eventually publishing influential books based on this research such as Social Life of Small Urban Places (1980).27
In the early twenty-first century, the new digital cultural universe made massive numbers of media artifacts and people’s online interactions easily available, and this attracted the attention of many researchers. But we have to remember that people have physical bodies that physical behaviors and cognitive and emotional processes are equally important parts of culture—even if their observation and analysis at scale may take more energy than observing the digital universe.
Among all types of cultural behaviors, one category is so important that we need to discuss it separately. In fact, this category is what separates our contemporary culture from earlier periods even more critically then scale. Surprisingly, it has been almost completely ignored in quantitative studies of culture carried out across many academic fields. This category is human-computer interaction.
The theoretical understanding of interaction and the choice of appropriate methods for its analysis go hand in hand. Before we rush to capture data, we need to ask: What is interaction, how it is created by particular interfaces, and how can it be represented as “data”?28 One of the twelve research challenges for cultural analytics formulated in the introduction presents this question: How do we use computational approaches to analyze interactive media and experiences (e.g., playing a video game, interacting with the Instagram app, experiencing an interactive installation), as opposed to only dealing with static media artifacts?
Consider the twentieth-century “atom” of cultural creation: a “document” or a “program”—that is, content stored in a physical form delivered to consumers via physical copies (books, films, audio records) or electronic transmission (television). In software culture, we no longer have documents. Instead, we have software performances. I use the word performance because what we are experiencing is constructed by software in real time. Whether we are exploring a website, playing a video game, or using an app on a mobile phone to locate nearby friends or a place to eat, we are engaging with the dynamic outputs of computation.
Although static documents and datasets may be involved in this interaction, you can’t simply consult a single PDF or JPEG file the way twentieth-century critics examined a novel, movie, or TV program. Software often has no finite boundaries. For instance, a user of Google Earth is likely to experience a different “earth” every time they use the application. Google could have updated some satellite photographs or added new street views and 3-D buildings. At any time, a user of the application can also load more geospatial data created by other users and companies.
Google Earth is not just an application. It is a platform for users to build on. And while we can find some continuity here with users’ creative reworkings of commercial media in the twentieth century—pop art and appropriation, music remixes, slash fiction and videos, and so on—the differences are larger than the similarities.
Even when a user is working only with a single local media file stored in their computer, the experience is still only partly defined by the file’s content and organization. The user is free to navigate the document, choosing both what information to see and the sequence in which to see it. For instance, in Google Earth, I can zoom in and out, switching between a bird’s-eye view of the area and its details; I can also switch between different kinds of maps.
Most important, software is not hardwired to any document or machine. New tools can be easily added without changing the documents themselves. With a single click, I can add sharing buttons to my blog, thus enabling new ways to circulate its content. When I open a text document in the macOS Preview media viewer, I can highlight, add comments and links, and draw and add thought bubbles. Photoshop allows me to save my edits in separate “adjustment layers” without modifying the original image. And so on.
All that requires a new way to analyze media and culture. What is interactive media “data”? Software code as it executes, records of user interactions (e.g., clicks and cursor movements), the video recording of a user’s screen, a user’s brain activity as captured by an EEG or fMRI? All of the above, or something else? To use terms from linguistics, rather than thinking of code as language, we may want to study it as speech.
Over the past fifteen years, a growing number of scholars in the digital humanities have started to use computational tools to analyze large sets of static digitized cultural artifacts, such as nineteenth-century novels or the letters of Enlightenment thinkers. Often they follow traditional humanities approaches—looking at the cultural objects, rather than peoples’ interactions with these objects. What has changed is the scale, not the method.
In my view, the study of software culture calls for a fundamentally different methodology. We need to be able to record and analyze interactive experiences, following individual users as they navigate a website or play a video game; to study different players, as opposed to using only our own game play as the basis for analysis; to watch visitors of an interactive installation as they explore the possibilities defined by the designer—possibilities that become actual events only when the visitors act on them.
In other words, we need to figure out how to adequately represent software performances as data. Some answers can come from the field of human-computer interaction (HCI), in which researchers in the academy and in industry study how people engage with computer interfaces. The goals of that research, however, are usually practical: to identify the problems in new interfaces and to fix them. Designers working in interaction design and game design study interaction as well—along with inventing new interfaces and interaction techniques.
The goals of cultural analytics for interactive media include not only quantitative but also theoretical analysis—understanding how people construct meanings from their interactions and how their social and cultural experiences are mediated by software. Therefore, though we can use methods of transcribing, analyzing, and visualizing interactive experiences that have been developed in HCI, interactive design, and game design, we may also need to invent our own.
The fourth category of cultural things that we can analyze algorithmically on large scale are cultural events, places and organizations. Because so many of them have some online presence or are organized via online services such as Meetup, we can collect, visualize, and analyze data about them.
Let’s look at some examples of sources of data in this category. Organizers of endless events around the world—discussions, festivals, concerts, exhibitions, competitions, conferences—create pages for their events on Facebook. Cultural organizations and projects also have their own dedicated websites. You can also start with particular cultural genres and check if there are sites listing large numbers of events worldwide for them. Examples include electronic music festivals, art biennales, design weeks, and fashion weeks. For instance, for the Elsewhere project, we assembled a list of art biennales by merging and checking information from a number of online sources (see figure I.1.)
Many cities have listings of their local cultural events and places. The most comprehensive such listing that I know of is Russian site Afisha (afisha.ru). It presents daily updated information about movie screenings, exhibitions, concerts, music performances, and theater events in almost two hundred cities in Russia. The site also lists cultural facilities (i.e., what I call places) and the programs of events there.
People around the world use dedicated online services to organize meetings, workshops, conferences, parties, and other gatherings. Examples of such services include Meetup (meetup.com), Eventbrite (eventbrite.com), and Timepad (timepad.ru). In March 2017, there were 272,000 Meetup groups in 182 countries, 608,036 monthly meetings, and thirty million members, and these numbers continued to grow.29 In the same year, Eventbrite was used to organize over two million events, with two million event registrations every week.30 Both platforms have APIs, so information about meetings and events, including group names, descriptions, categories, dates, and geographic locations, can be used in research. For example, browsing the list of groups organized by topic on Meetup for September 21, 2019, I find 935 Alternative Energy groups (288,273 members), 3,640 Environment groups (1,467,963 members), and 22,766 Meditation groups (8,656,762 members). Exploring the data for 2,635,724 Meetup group meetings from 2004 to 2019 we collected for the Elsewhere project showed the geographic reach of such platforms: these meetings took place in 17,360 cities in 146 countries.
Of course, we have to remember that such services are not equally popular in every country, and this has to be taken into account in conceptualizing projects and analyzing data. The services available only in English may have more listings from English-language countries. Some services are only used in their countries. For instance, earlier I evoked Behance as an example of broad geographic participation in cultural portals. But although our sample of 81,684 Behance accounts created from 2007 to 2019 collected for Elsewhere includes cities in 162 countries, predictably, the two countries with the most accounts were the United States (16.6 percent) and the United Kingdom (6.7 percent).
Meetup, Eventbrite, and Behance have a higher proportion of participants from the United States and/or other English-language countries, but the opposite is true for Facebook. It better represents cultural and intellectual life in the rest of the world rather than in the United States and Western Europe. Many academics, artists, and intellectuals in the West started to leave Facebook after 2016. The reasons for this are multiple: they may not like companies perceived as monopolies; they may believe that by participating, they are doing free labor for Facebook; they may not like that their data is used for personalized ads; they may have been influenced by many negative media stories, such as coverage of Cambridge Analytica’s work on the US presidential campaign and the Brexit vote.
Meanwhile, the number of global active users on Facebook has increased every year, reaching 2.5 billion monthly active users by end of 2019. The countries with the most active monthly users at that time were India (260 million), the United States (180 million), Indonesia (130 million), Brazil (120 million), Mexico (84 million).31 US users only accounted for 7.2 percent of active users worldwide. While people in the United States and Western Europe have access to a variety of older communication and publication platforms, in many other countries Facebook is the only viable and free platform for intellectual and cultural communication and is used by the majority of intellectuals, academics, artists, NGOs, and cultural groups. In some countries, all video and media artists only have YouTube and Vimeo as their exhibition platforms.
One reason for this is the existence of censorship and government surveillance (real, imagined, or both) that pushes cultural and intellectual life and communication to global social networks and messaging apps. For example, in 2019, intellectuals in Russia were publishing whole journals on Telegram (an instant messaging service), while posts of Russian users on Instagram often included long reflective texts.
Another reason is that people and organizations in developing countries are more open to new technologies because their populations are younger or because these countries started to grow economically in the web era. For example, I am often contacted by cultural organizations, museums, festivals, and universities from many countries—and these contacts and all subsequent correspondence would take place over Facebook Messenger (or WeChat in the case of China) for developing countries, while similar organizations in the United States and Europe would contact me via the older technology of email. (And I even still get some invitations though regular mail from Europe.) Trends in the general use of social networks around the world show similar differences. In the Visual Earth project, we used a unique dataset of 270 million geotagged images shared on Twitter around the world between September 2011 and June 2014. We looked at correlations between the growth of image sharing on Twitter and a few economic and demographic indicators—and the strongest correlation was with the median age of a country’s citizens (-0.73). In other words, the younger the average age of a country, the faster the growth of image sharing from 2011 to 2014. This correlation was even stronger than between growth rate and economic development (-0.52).
As we learned, there are many online sources for data about cultural events, organizations, and professionals. They include millions of dedicated sites for academic conferences, educational programs, and cultural centers; large platforms such as as LinkedIn and Behance; and general social networks in which most cultural events are advertised (e.g., Facebook pages). What all these data sources have in common is that their data is in formats that are relatively easy to analyze—dates, geographical locations, and categories (e.g., the number of artworks a museum has for a given century, in a particular medium, or from a particular country). These are all instances of structured data, which can be easily visualized and also explored using well-known descriptive statistics methods. In addition to structured data, such sites have text data—names of events and organizations and their descriptions. The methods for text analysis are also well developed, and there are many tutorials and free textbooks online explaining how to perform such analysis using various programming languages. (The field of computer science that develops and tests these methods is called natural language processing. For examples of using this approach for literary and historical texts, look at textbooks and publications in the digital humanities and digital history.)