As we saw in the previous chapters, we learn from games—how to play them and almost certainly other skills as well. We can also develop algorithms that learn to play games. But let’s turn this statement around. Can games learn from us? And if so, what could they learn? Can we develop algorithms that use our interactions with games to learn about us?
When you play a game, you are constantly supplying information to the game. You are pressing buttons and twiddling console sticks. In many games, you are also entering text. You are constantly making choices: go this way or that way, respond affirmatively or negatively to that character in a conversation, attack that enemy or not (and using which weapon). Some choices are complex and expressed over the course of a whole game, such as the personality and other characteristics of the character you are playing or the shape and political orientation of the country you rule; others happen at subsecond scales, such as exactly when to jump off a platform to avoid falling into a gap.
All of this is information that can be nicely expressed with numbers and other symbols. For games implemented on computers (including standard computer games as well as digital versions of board and card games), this is convenient because that’s what computers are great at: storing and processing information. It’s perfectly possible for a computer game to store all the input you have ever given it and then use clever algorithms to analyze it. These days, almost all devices we play games on (computers, smartphones, game consoles) are connected to the Internet. Given an Internet connection, it is perfectly possible for a game to “phone home” and send all the data it has gathered from you as you played it, either in raw form or aggregated, to the servers of the company that made the game. The game developer can then run all kinds of algorithms on the data to find out things about you and the rest of its player population. In fact, very many—maybe most?—recent video games already do this.
But what kinds of things can games learn from you?
What Would You Do?
Just like you can learn from a game how to play it, the game can learn from its players how it is played. By looking at the histories of how players have played the game, it is possible to find out what players typically do in each situation. This information can be used to create an AI that plays the game like an “average” player by simply taking the most frequent action in each situation. To see how this can be done, imagine that the game simply stores a long list of all the situations the player has ever been in (in the game) and the action the player took in each situation. Let’s assume we can describe the situation with some numbers; for example, the coordinates of the player in the game world, current health, the relative position of the closest nonplayer characters (NPCs), and so on. After we’ve stored all these data in a long list, it becomes trivial to create an AI agent that can play the game just as the human player would. At every point in time, simply look at what situation the agent’s character is in, find that situation in the long list of situations the player encountered, and take the action that the player took. Simple and elegant, right?
There are two problems with this simple solution. The first is that the list of all the situations a player has encountered can grow long—very long if you record where the player is, say, ten times per second and the player plays for ten hours, and very very long if you want to learn from not just one player but perhaps hundreds or millions. The length of the list is a problem not just for storing it in computer memory, but also for being able to look up one of these situations quickly. You don’t want to look at millions of different stored actions every time you want to figure out what to do. We need a more compact way of storing the complete playing history of a player (or multiple players).
The other problem is that even if you spent ten hours playing a game, you have almost certainly not experienced every possible situation in that game. In fact, even if you have hundreds of players in your list, you are going to be missing lots of potential situations in your list. For every game that is not entirely trivial, the number of possible different game states is going to be some insanely large number, probably bigger than the number of stars in the universe.1 You’ll also need a way for your agent to deal with situations that the player(s) did not encounter. So we need a way to generalize.
Luckily, it turns out that you can use the backpropagation algorithm to train neural networks to predict what the player would do. Yes, this is the same method I described in the previous chapter when I talked about learning how to drive a car through trial and error. The difference is that here, we are using backpropagation not for reinforcement learning but for supervised learning. In supervised learning, you have a list of “instances,” where each instance has a number of features that describe different aspects of the instance, and a target value. When learning to play the game like a human, each instance would be composed of the features describing a situation the player agent was in and what action the player took in the situation. Backpropagation is then used to train the neural network to reproduce this list. Remember that in reinforcement learning, the backpropagation algorithm changes the weights of the neural network depending on whether the action the network decided on leads to good or bad outcomes; in supervised learning, it changes the weights depending on whether the action the neural network decided on was the same as what the human decided on. Using this simple principle, the neural network can be trained to predict what action the player would have taken in each situation, usually with very good accuracy. The great advantage of this is that the neural network is much smaller than the long list of situations and actions used to train it, and it’s much faster to “ask” the neural network for an action than it is to look up the state in a big table. Such neural networks typically also have a pretty good ability to generalize, meaning that they can come up with an answer for what the agent would do in a situation that the player never actually encountered based on what the player did in similar situations.
Who Are You in the Game?
For a game developer, it is crucial to know who plays their game: which aspects of it they are good and bad at, which aspects they like and dislike, and generally what they will do in the game. Outside the world of games, marketers use terms such as target group analysis and market segmentation when they talk about identifying and characterizing the potential customers of a product, so that the company that makes the product knows how to sell it or improve it. In games, we talk about player type analysis. The idea is that players of a game can be clustered into different groups, or player types, where the players of each type behave similarly and have similar preferences. An early and very influential attempt to identify player archetypes was made in the 1980s by Richard Bartle, a pioneer of online multiplayer games. Bartle built on his observations of players in the text-based online game MUD and stipulated four player types: achievers, who like to accumulate points and get ahead in the game; explorers, who like to explore both the space of the game and the rule system and find new places or invent new ways of playing; socializers, who are attracted to online games because of the opportunities to hang out with and talk to others; and finally, killers, who enjoy causing harm to other players’ in-game characters.2
Obviously this typology works best for the type of games it was devised for: online multiplayer games. While the categories of achiever and explorer would be easy to apply to Super Mario Bros. and Angry Birds (and maybe Chess, though it is unclear what exploration means in such a game), the categories of socializer and killer make no sense for one- or two-player games. It is likely that you would need to find a different typology for each game or at least for each game genre. Fortunately, we have the tools to do this now, given all the data that games collect about us and modern data processing and machine learning techniques. In other words, games can learn player typologies from players.
In 2009, some of my colleagues at the IT University of Copenhagen (ITU), Alessandro Canossa, Anders Drachen, and Georgios Yannakakis, managed to get hold of a treasure trove of player data. Through a collaboration with the video game publisher Square Enix Europe, they gained access to data collected from about a million players playing Tomb Raider: Underworld on the XBox 360. The games in the Tomb Raider franchise are action-adventure games in which you play a single character (the adventurer Lara Croft) and navigate a three-dimensional world while solving puzzles and fighting bad guys and occasionally monsters (figure 6.1). The developers had included functionality in the code so that every time a player finished a level, the game contacted Square Enix’s servers and uploaded a chunk of information about how the player had played the level. This information included how much time the player character had spent in various parts of the level, how many treasures found, how many enemies killed, and how often the player used the game’s help system, among other things. This was a new and untested idea in 2009 (in 2018, it would be hard to find a commercially released game that does not “phone home” to the developer with information on how it’s being played), and therefore the data were rather dirty and a lot of work was needed to get them into such shape that machine learning algorithms could be used on it.
Figure 6.1 Balancing on a ledge in Tomb Raider: Underworld (Crystal Dynamics, 2008).
The cleaned-up and reorganized data were fed to an algorithm known as a self-organizing map. This is a type of neural network. Like the ones discussed in the previous chapter, it is a computational structure inspired by the human brain, but it works rather differently from the car-driving networks discussed there. A self-organizing map takes a large amount of data and separates the instances into different groups so that the instances within one group are maximally similar and instances in different groups are as different from each other as possible. In machine learning language, this is called clustering and is a form of unsupervised learning (as opposed to supervised learning or reinforcement learning). You don’t know in advance how many groups you are going to get; this depends on the data and, to some extent, how you have configured the self-organizing map. In this case, each instance represented one player and contained carefully selected information about what sorts of things the player had done over the course of the game. Out came four clusters of data, representing four player types.
Simply knowing that there are four types of players doesn’t tell us much. As a developer, we would want to know what those player types represent—in other words, how the players of one type differ from those of another. So the team looked at a number of representative players of each type and compared how much they had done each kind of action. They identified four types: veterans, who rarely die, collect most of the treasure, and generally play the game very well; solvers, who rarely use the help system or any hints, play slowly, and prefer to solve all of the game puzzles themselves; runners,who complete the game very quickly but frequently ask for help and tend to die more often; and pacifists,who are good at solving the game’s puzzles but are bad at the combat elements and appear to seek to avoid them. This typology is clearly very different from Bartle’s, which is understandable given that we are dealing with a very different type of game with a different player population. Something that is rather interesting is that the developers of the game at Square Enix had not foreseen the existence of the pacifist player type when they developed the game, and they were surprised to find out that the game was played in a way they had not “intended.”3
While it is obviously useful to know what types of players play your game, it would perhaps be even more useful to know what the players are going to do in the game. Usually you want your players to stay with your game for as long as possible, because a happy player will recommend your game to a friend and perhaps buy your next game. It is also common with free-to-play games that are initially free but involve semi-mandatory payments for upgrades in order to keep playing. For developers of such games, it absolutely essential to be able to predict which players will stay with the game (and eventually pay money) and which might stop playing it. Why? Because when you know which aspects of your design make people stay and pay, you can tweak your game to make more money. In addition, as a game developer, you may simply be interested in understanding your players.
The next task for the same team, of which I was now a member (I had just moved to ITU to take up my first faculty position), as was Tobias Mahlmann (one of our PhD students), was to try to learn rules that would predict player behavior later in the game from player behavior early in the game. One of the things we tried to learn to predict was the highest level a player would complete out of the seven levels in the game. Theoretically, there are many supervised learning methods that could be used to learn to predict this, but some are better suited than others. We tried several of these methods on the task of predicting after which level the player would stop playing. One of the best-working methods was decision tree induction, a method that also has the advantage that its results are easy to understand for humans. It produces decision trees, which can be thought of as long lists of if-then rules within each other. Here is an example of what the algorithm learned:
IF Rewards on level 2 <18.5
THEN IF Time in Flushtunnel <9858: 2
ELSE (Time in Flushtunnel ≥9858): 3
ELSE (Rewards on level 2 ≥18.5): 7
In other words, if you accumulated a low score on level 2 and spent little time in the Flush Tunnel (an area in level 2), you will stop playing after level 2 and never finish level 3. Otherwise you will stop playing after level 3. However, if you accumulated a high score on level 2, you will finish the whole game.
This no doubt sounds like a very silly, arbitrary rule. It looks about as reasonable as astrology, and it’s not the kind of rule you would expect an actual human game designer to come up with. However, silly as it may be, it is built on solid empirical evidence: it has a prediction accuracy of 76.7 percent when tested over tens of thousands of players. This means that while there are a certainly a few people who get a low reward on level 2 and then continue to finish the whole game, it is statistically unlikely. While it might be insulting to common sense that the amount of time spent in some tunnel should be so indicative of whether a player will give up the game after level 2 or 3, this really seems to be the case based on all these data. Maybe the most noteworthy result is that the prediction accuracy is so high. What this says is that we humans really are quite predictable, even when we play games.4
Who Are You Outside of the Game?
So far we have seen that the game can learn from your playing what type of player you are and how you will play in the future. But you are not only a player of games. You are a full-fledged human being, with hopes, dreams, fears, manners, friends, and habits. There is no reason to believe that all the rest of who you are disappears the moment you lean back on the sofa and grab the Xbox controller; you are still you, even if you are momentarily Mario, Master Chief, or Lara Croft. Now the question is, Does anything of the rest of you shine through in your game playing? What can the game learn about the real you from analyzing how you play?
Back in 2013, Alessandro and I had at our disposal an ambitious master’s student, Josep Martinez, and we were searching for a topic for his thesis. Alessandro had recently read the works of Stephen Reiss, a personality psychologist who had devised a model for categorizing people’s life motives, that is, what motivates them in life. Reiss identified sixteen broad life motives (in alphabetical order): acceptance, curiosity, eating, family, honor, idealism, independence, order, physical, power, romance, saving, social, status, tranquility, and vengeance. Each of these motives has several subcategories, and there is a well-tested questionnaire available for assessing life motives. We wondered whether the motives people had in real life were also expressed in games. If so, which ones? And in which games?
Like so many others, Alessandro, Josep, and I were fascinated by Minecraft, the open world game that took the world by storm beginning in 2010. When it was first released, as a buggy beta, Minecraft was a rather unique game—now there are many clones—not only for its peculiarly blocky graphics but also for the unparalleled freedom it affords players. The game is now a global phenomenon that is used for everything from making machinima (animated films made inside video games), to education, to testing AI algorithms. Minecraft can be described as a cross between a role-playing game and a digital version of Lego (figure 6.2). When you arrive in the game, you have nothing, and you must hurry to assemble some tools so you can build yourself some shelter before the night comes and monsters start roaming the land. But in order to make these tools, you need materials, and in order to get those, you need to mine the ground. After crafting more advanced tools, you will be able to mine deeper for more exotic materials so you can construct more advanced buildings and mechanisms. Given enough time and effort, you can construct anything you want. Searching for videos of Minecraft on YouTube yields thousands of examples of player-constructed replicas of famous buildings and vehicles (even the Starship Enterprise). There is also a storyline in Minecraft, including fairly typical role-playing game-like quests, but it is entirely optional whether to follow this storyline and carry out the quests; many players don’t.
Figure 6.2 The cubistic world of Minecraft (Mojang, 2011). The game largely revolves around mining cubes for material so that you can build things out of other cubes.
Almost all games afford a number of different playing styles, but Minecraft does so more than most others. I think it’s safe to say that there are more different ways to play Minecraft than there are ways to play Tomb Raider: Underworld. Clearly these different playing styles reflect different in-game motivations: some people are motivated by finishing quests, others by expressing themselves through building grandiose edifices, yet others by collecting rare resources. But do these motivations have anything to do with your real-world life motives? Does someone who cares mostly about her family play differently from someone whose chief concern is getting ahead in professional life? We decided to find out.
Josep sent out questionnaires, with questions taken from the Reiss Motivation Profile, to 100 Minecraft players; these questionnaires were used to construct a profile of each player in terms of what motivated them most. He then asked each player for her Minecraft log file. This is a small file automatically saved by the game, which contains more than six hundred variables, including such things as how many hours the player has played, how much redstone ore she has mined (redstone is used for making electric-like circuits), and how far she has traveled by pig (an often overlooked transportation option). After extracting and cleaning these data, we ran a correlation analysis of all combinations of potentially relevant game variables and life motives.
“Correlation” is a way of saying that, statistically, two things have something to do with each other. It does not necessarily mean that one causes the other: if umbrella sales and the number of hours you spend watching TV are correlated across the weeks in a year, they might both be caused (at least in part) by bad weather. Two variables can correlate negatively or positively. So is, for example, smoking negatively correlated to longevity: when one is high, the other is low. (In this case, it is reasonable to assume that one causes the other.)
We found that all of the life motives were significantly correlated with several of the in-game variables. However, some were correlated with only a few variables (so few that it might come down to chance), whereas others were correlated with a large number of the in-game variables, and some of the correlations were so strong that there was virtually no room for doubt. Among the most highly correlated life motives were curiosity, saving, vengeance, and honor, whereas those that did not seem to be much expressed in the game were romance, tranquility, and physical activity. In some cases, these correlations make intuitive sense to someone who knows the game; in other cases, they are unexpected and quite amusing. People who are strongly motivated by curiosity in real life tend to craft plenty of torches and stone tools in the game, which makes sense because these are the most cost-efficient ways of exploring large parts of the game world. Those who are motivated by saving tend to use cheap and simple materials in the buildings and tools they construct. Vengeful players apparently quit the game and restart it (perhaps from an earlier save) more often—what would be called “rage quit” in gamer lingo. Players who are strongly motivated by independence in real life showed this in the game by refusing to do the quests in the game’s storyline; in particular, it was strongly correlated with not even attempting the final quest. Another interesting expression of life motives is that people with a strong need for tranquility built significantly more fences around their dwellings. It certainly seems that the person you play when you play Minecraft is you in some very important respects.5
Figure 6.3World of Warcraft (Blizzard, 2004) is a massively multiplayer online role-playing game; much of the game is communicating with other players over text or voice chat.
These results can be seen in the light of the studies by Nick Yee, then at Stanford University, and his colleagues, who investigated how players express their personality (rather than life motives) in the online multiplayer role-playing game World of Warcraft (figure 6.3). Yee used the Big Five personality questionnaire, which groups personality traits into the five categories of Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. There were plenty of correlations in these data as well, and he could see, for example, that conscientious players were more likely to collect items of various kinds and less likely to die of accidents, that players with high openness explored more of the game world, and that extraverted players (unsurprisingly) had more social interactions in the game.6 A group led by Pieter Spronck and including Shoshanna Tekofsky at the University of Tilburg has also found similar effects in games as different as the epic strategy game Civilization and the first-person shooter Battlefield 4. For example, it’s possible to predict gender and age with relatively good accuracy from how people play Battlefield 4.7
Taken together, the picture we get from this research, as well as many other studies on these topics, is that you express quite a lot of yourself while playing games. If the game wants to, it can find out not only who you are in-game and how you will play in the future, but also quite a lot about who you are outside of the game. This raises plenty of interesting opportunities not only for game developers but also for psychologists and other social scientists who want to understand how humans function.
But this research also raises a number of complicated questions. A couple of years ago, I was speaking at a conference where a number of people from the security services and other government agencies were in attendance. One of the things I talked about was how much you could find out about players from their in-game behavior. In order to stir the conversation, I suggested that it might be possible to find out really sensitive information about players, such as their political views, sexuality, history of drug use and incarceration, or health status. (Note: I have not done this research and do not intend to!) I was expecting to get some worried reactions, but instead these people simply nodded pensively, as in “that’s an interesting idea.” It is fair to say that I have not become less worried about the potential to use player modeling for nefarious purposes since then. Particularly in light of concerns about how much of our personal information is gathered by security services, social network companies, internet providers, and all manners of shady operators who sell their services to the highest bidder, I think it is important to realize that our game playing is another way in which we leave rich digital trails. The difference, perhaps, is that when we post on a social network, we are aware that we are sharing information about ourselves; when we play a game, this is not obvious because we believe we are only acting inside the game world. But as we have seen, we bring much of ourselves into that world.
