Chapter 3. Planning and Running a Manual Game Day

How many times have you heard comments like the following after a production incident?

“We were not prepared for that!”

“Dashboards were lighting up, we didn’t know where to look...”

“Alarms were going off, and we didn’t know why...”

The absolute worst time to try and learn about the weaknesses in your sociotechnical system is during an incident. There’s panic; there’s stress; there may even be anger. It’s hardly the time to be the person that suggests, “Shall we just step back a moment and ask ourselves how this all happened?” “It’s a little too late, it’s happening!” would be the reply, if you’re lucky and people are feeling inordinately polite.

Chaos engineering has the single goal of helping you collect evidence of system weaknesses before those weaknesses become incidents. While no guarantees can be made that you’ll ever find every one of the multitude of compound potential weaknesses in a system, or even just all the catastrophic ones, it is good engineering sense to be proactive about exploring your system’s weaknesses ahead of time.

So far you have hypotheses of how your system should respond in the event of turbulent conditions. The next step is to grab some tools and start breaking things in production, right? Wrong!

The cheapest way1 to get started with chaos engineering requires no tools. It requires your effort, your time, your team’s time, and ideally the time of anyone who has a stake in your system’s reliability. Effort and time are all you need to plan and run a Game Day.

What Is a Game Day?

A Game Day is a practice event, and although it can take a whole day, it usually requires only a few hours. The goal of a Game Day is to practice how you, your team, and your supporting systems deal with real-world turbulent conditions. You collectively get to explore:

• How well your alerting systems perform

• How your team members react to an incident

• Whether you have any indication that your system is healthy or not

• How your technical system responds to the turbulent conditions

A Game Day is not just a random event in which things break; it is a controlled, safe, and observed experiment through which you can collect evidence of how your sociotechnical system responds to the turbulent conditions that underpin your chaos engineering hypothesis. At its most basic it doesn’t require any tools at all—just a plan, a notepad and pen, and an awareness of what is going on and just how much you can learn from it!

A Game Day turns a hypothesis from your backlog into an experiment you can execute with your team to rapidly surface weaknesses in your entire sociotechnical system. You deliberately practice for unexpected conditions and capture evidence of any weaknesses in how you, your team, and your supporting systems deal with those conditions, choosing what to learn from and improve over time.

Planning Your Game Day

A Game Day can take any form you want as long as it ends up providing detailed and accurate evidence of system weaknesses. As you gain experience planning and running Game Days, you are likely to come up with all sorts of ideas about how you can improve the quality of your findings. For your first few Game Days, the following steps will help you get started successfully:

1. Pick a hypothesis to explore.

2. Pick a style.

3. Decide who participates in your Game Day and who observes.

4. Decide where your Game Day is going to happen.

5. Decide when your Game Day is going to start, and how long it will last.

6. Describe your Game Day experiment.

7. Get approval!

As you work through this chapter, you will build and then learn how to execute a Game Day plan—but before you can get started, you’ll need to pick a valuable hypothesis to explore during your Game Day.

Pick a Hypothesis

Trying to decide what to explore in a Game Day can be a pain if all you have to go on is what people tell you worries them. Everyone will have different pet concerns, and all of those concerns are valid. As a Game Day is not inexpensive in terms of time and effort, you will feel the pressure to ensure that the time and attention are spent on something as valuable as possible.

If you created a Hypothesis Backlog earlier (see “Creating Your Hypothesis Backlog”), then you’re in a much better place. The hypotheses in your ever-changing and expanding backlog make it as easy as possible to collectively decide what to invest a Game Day in, as each one describes:

• How you hope the system will respond

• The turbulent conditions that could be involved (failures, traffic spikes, etc.)

• The collective feel for how likely these turbulent conditions might be

• The collective feel for how big an impact these conditions might have on the system (although the hypothesis should be that the system will survive in some fashion)

• The valuable system qualities that you hope to build trust and confidence in by exploring the hypothesis

Together with your team and stakeholders, you can use these hypothesis descriptions to reach agreement on one that is valuable enough to explore in a Game Day. Then it’s time to consider the type of Game Day you are looking to perform.

Pick a Style of Game Day

Game Days come in an ever-increasing set of styles and formats. They tend to vary in terms of how much prior information is shared with the participants, and thus in what the scope might be for surfacing evidence of weaknesses in the system. A couple of popular Game Day styles are:

Dungeons & Dragons

Where none of the participants are aware of the conditions they are walking into

Informed in Advance

Where the participants are told before the Game Day about the type of incident they are walking into

Picking a Game Day style is not just a question of taste. It’s important to consider what evidence you are likely to find if applying one style versus another. For example, an adversarial Dungeons & Dragons–style Game Day in which no one is aware of the actual problem beforehand will explore:

• How the team detects turbulent conditions

• How the team diagnoses turbulent conditions

• How the team responds to turbulent conditions

An Informed in Advance–style Game Day will likely be limited to showing how the participants respond to turbulent conditions.

Decide Who Participates and Who Observes

Deciding who will attend is one of the most important steps in planning your Game Day, but possibly not for the reason you’d expect. You might think that the more important decision is who is going to participate in your Game Day, but that’s usually straightforward: you invite your team and anyone else who would likely be involved if the turbulent conditions of your hypothesis were to happen “for real.” Add all of those names to your “Invite List”—see Figure 3-1.

Figure 3-1. Game Day participants and observers list

Then it’s time to consider the second—and just as important—group for your Game Day: the people who are going to observe it.

This list should include everyone who might have any interest in the findings. Think broadly when compiling your observers list, from the CEO on down! You are trying to increase the awareness of the findings for your Game Day.

Decide Where

There are two things to decide when it comes to the question of “where” of your Game Day:

• Where will everyone be?

• Where will the turbulent conditions occur?

Ideally, the answer to the first question is “where they would be if the actual conditions occurred for real.” The second question is harder to answer. You could conduct the Game Day experiment by applying the turbulent conditions in production, but…you don’t want this Game Day to accidentally turn into a real incident!2

Often Game Day experiments limit their Blast Radius (the potential real-world impact of the experiment) to a safer environment than production, such as a staging environment. Nothing beats production for giving you the best possible evidence of real weaknesses, but if you run a Game Day and you take down production, that just might be the last bit of chaos engineering you’ll be asked to do!

Running Your Game Day Somewhere Safe(r)

There’s no doubt that performing Game Days—even the automated chaos experiments that you’ll learn about in Part III—in production, where there are real users, is how you are likely to learn the most. It’s also possible that some experiments can’t be simulated anywhere else due to conditions and scale needs.

However, it’s usually best to start executing your experiments in a safer environment with a small Blast Radius and then to scale up gradually with larger and larger Blast Radii, gaining trust and confidence in your system’s ability to handle the conditions, before you go anywhere near production with the experiment.

Decide When and For How Long

A Game Day experiment can take up an entire day, but don’t be afraid to make it a lot less than a whole day either. Even just three hours is a long time to be in a crisis situation, and don’t think that there won’t be huge amounts of stress involved just because your participants know the issue is in a “safe” staging environment.

You should plan your Game Day for when you anticipate that the participants and observers are going to be most open to, and even eager to learn from, its findings—for example, before a team retrospective.

If your team does retrospectives regularly, then it’s often a great idea to execute your Game Days just before them. This very neatly helps to avoid the problem of having dull retrospectives—nothing beats dullness like walking into a retrospective armed with an emotionally fraught set of evidence of system weaknesses just experienced during a Game Day!

Describe Your Game Day Experiment

Now it’s time to construct your Game Day experiment. The experiment will include:

A steady-state hypothesis

A set of measurements that indicate that the system is working in an expected way from a business perspective, and within a given set of tolerances

A method

The set of activities you’re going to use to inject the turbulent conditions into the target system

Rollbacks

A set of remediating actions through which you will attempt to repair what you have done knowingly in your experiment’s method

So far you have a backlog of hypotheses, but the emphasis of a steady-state hypothesis is a little different from the hypothesis card shown earlier in Figure 2-7.

“Steady-state” means that you can measure that the system is working in an expected way. In this case, that normal behavior is that “the system will meet its SLOs.” You need to decide what measurement, with what tolerance, would indicate that your system was in fact responding in this timely fashion. For example, you could decide to probe the system on a particular URL and ensure that the system responds with a tolerance that included an expected HTTP status code and within a given time that was set by a Service Level Objective.3

There might be multiple measurements that collectively indicate that your system is behaving, within tolerance, in a normal way. These measurements should be listed, along with their tolerances, as your Game Day’s steady-state hypothesis—see Figure 3-2.

Figure 3-2. Chaos experiment steady-state hypothesis

Next, you need to capture what you’re going to do to your system to cause the turbulent conditions, and when you are going to perform those actions within the duration of your Game Day. This collection of actions is called the experiment’s method, and it should read as a list of actions that cause all the failures and other turbulent conditions you need to create for your Game Day—see Figure 3-3.

Figure 3-3. Chaos experiment method

Finally, you capture a list of remediating actions—or rollbacks as they are usually called—that you can perform to put things back the way they were, to the best of your ability, because you know you caused problems in those areas when you executed the actions in your experiment’s method (see Figure 3-4).

Figure 3-4. Chaos experiment complete with rollbacks

Get Approval!

Last but never least, make sure that you create a list of people who will need to be notified and approve of running the Game Day. This could be everyone from the CEO to up- or downstream systems, and on through to third-party users of the system. Add to your Game Day plan a “notifications and approvals” table (see Figure 3-5) and make sure you’ve chased everyone down before the Game Day is announced.

Figure 3-5. Table of notifications and approvals

Running the Game Day

Your job when running your Game Day is to adopt the role of “Game Day facilitator.” This means that you aren’t a participant. Your job is to keep a detailed log of everything that happens so that this record can be used to elicit weaknesses after the event. Feel free to use every piece of technology or trick you can think of to record as many observations as possible during the Game Day; you never know what might lead to an interesting finding. For example, a participant staring off into space for 20 minutes might appear to be of no interest, but if you make a note of it and then ask them about it afterwards, you might find out that they were thinking, “I don’t even know where to start to look,” which is valuable information.

Consider a “Safety Monitor”

Sometimes you might be facilitating a Game Day experiment against a system that you have little personal knowledge of. This tends to happen when you are asked to help another team begin to adopt chaos engineering. If this is the case, be aware that you might need a “Safety Monitor.”4

A Safety Monitor is likely the most experienced expert on the system that is being used for the Game Day. You have to accept the compromise that they will not be a participant but will be working closely with you, the facilitator, instead. Their job is to alert you if the Game Day is going in a dangerous direction.

For instance, imagine that you planned for the Game Day to be exercised in a safe staging environment, but the team accidentally starts to diagnose and manipulate production. The Safety Monitor, with their expert knowledge of the system, will likely detect this sort of dangerous deviation and can advise you to halt the Game Day before any damage occurs beyond the expected Blast Radius of the experiment.

The Case of the Lucky Game Day Facilitator

I was feeling great. I suspected (strongly) that there was a weakness in the system, and his name was “Bob.”⁵ Bob was a hero, and it was easy to spot. Ask anyone on the team how something was done in production, or even in staging, and the answer was usually “See Bob,” or at least “Better see Bob to check.”

Many teams have Bobs—heroes that end up being single points of knowledge and failure. It’s not always their fault, either; sometimes they are simply the person who has been around the longest, or the one who was able to most quickly build a mental model of the system. Bobs happen all the time, and you probably know of one in your own team—but this doesn’t take away from the fact that Bob is a weakness, because he (or she) is a single point of knowledge.

Now I wanted proof. I wanted to make this potential weakness in the system unignorable by everyone. I could just tell everyone that Bob was a single point of failure, but that tactic more often than not results in a shrug and a “Yeah, we should do better at that…” I wanted people to feel the problem, and the best way to explore this was with a chaos engineering Game Day.

So I asked Bob’s boss if I could kill Bob. In fact, the question I posed was, “Can Bob go under the proverbial bus?” It may sound nasty, but I was simply asking whether I could run a Game Day in which Bob was totally and utterly out of action. Not present, virtually or otherwise—complete disconnection. After I clarified to Bob’s boss that I wasn’t actually looking to commit murder, the plan was agreed to.

When I spoke to Bob, he was on board with the plan, with one condition: I had to phone his wife and explain that he would be completely out of contact for the three-hour Game Day. I’d made it clear that he had to disconnect because there was every chance the team might resort to calling him, so it had to look like a complete shutdown of all communications with Bob for this to work. Fortunately, Bob’s wife was quite keen on him “going under the bus,” which I chose not to explore with any more questions.

The Game Day was planned and scheduled. Bob took his place in an adjoining room with no connectivity. I introduced some turbulent conditions in the staging environment and briefed the team that we were doing a Game Day against staging. The first reaction from the team was perfect: “We can’t do the Game Day, Bob isn’t here.” Yes! That’s just what I expected, and I gently encouraged everyone to continue regardless.

Now, let’s skip to the middle of the Game Day. It was around two hours in, and the team was clearly struggling but making ground. They didn’t know where the playbooks were (noted as a weakness), and they had a nightmare trying to get hold of passwords so that they could start probing the system to figure out what was going on (also duly noted); however, they seemed busy, and the atmosphere was one of progress.

But I was nervous. I’d seen a lot of actions being taken without consulting the playbooks or any other reference material. Commands were often being executed with a precursory muttering of “Let’s just try this…” That might have been fine, but I was starting to feel nervous about the limits of my own knowledge of the system. I was worried about something being done that I wasn’t smart enough to stop.

I was starting to feel that even I needed Bob.

So I began popping over to the adjacent room and running some questions by Bob. I mentioned some commands and parameters being used and asked whether these sounded at all right. Then before I could continue further with my questions, Bob leaped off his seat, barreled past me, and announced with a shout of horror, “Stop!”

The Game Day was suspended immediately. It turned out the team had logged into the wrong system—production, in fact—and were in the process of destroying tables in the production database. They had no idea they’d headed to the wrong target, and were flummoxed as to why the system seemed “fine” when the dashboards were actually reporting turbulent conditions. Fortunately for me and the team, I’d followed my gut and spoken to Bob in time to stop major damage being done.

We learned a lot from this Game Day—the list of observations of weaknesses in knowledge, practices, tooling, playbooks, etc. just went on and on. But the biggest lesson of all was for me. Bob had been my impromptu “Safety Monitor,” and I’d never again conduct a Game Day without a Bob at hand.

Oh, and Bob was indeed a single point of failure, and everyone had felt that pain. In the ensuing retrospective, dedicated plans were put in place to begin to increase everyone’s knowledge of the runtime system, with Bob’s full endorsement. In fact, he’d been mentioning for ages how he’d like to train people up, but only now was the motivation there for this to actually happen. One of the observers of the Game Day was the product owner, who was only too happy to dedicate the time and budget to this as well.

Mission accomplished—but perhaps a much riskier lesson to learn than I would have liked.

Summary

In this chapter you learned how to build and run your first chaos experiment as a manual Game Day. Game Days are very useful for exploring weaknesses collaboratively. If the Game Day is properly planned, the findings are hard to ignore, and you can achieve much with this relatively small investment in time and effort. Game Days are also perfect for exploring the human side of the system, which is where a huge amount of system weaknesses are often found.

However, Game Days are cheap in tooling but not in time and effort, so there are limitations when it comes to how often you can plan and run them. Each Game Day will help to build more trust and confidence in how your sociotechnical team will react to turbulent conditions, but if you can execute a Game Day only once a week, month, or quarter, then that’s the limit of how often you can learn from weaknesses. In the next part of the book, you’re going to learn how you can overcome these limitations of Game Days by creating automated chaos experiments.

1 Cheapest in terms of tooling, not in terms of time.

² See “Consider a “Safety Monitor”” for how this can happen even when you don’t plan to use the production environment on Game Day.

³ Find more on how to set SLOs in Site Reliability Engineering, by Niall Richard Murphy et al. (O’Reilly).

⁴ You might need one regardless; in fact, always consider the possibility.

⁵ Yes, the name has been changed to protect the innocent.