4. The Purpose of Scrum
Often, a school is your best bet—perhaps not for education
but certainly for protection from an undead attack.
—Max Brooks, The Zombie Survival Guide
In This Chapter
• Discover how Zombie Scrum compares to Scrum as it is intended.
• Understand the underlying purpose of the Scrum Framework, and how Scrum is all about navigating complex problems and reducing risk.
In our frantic search for an antidote to Zombie Scrum, we looked on the other side of stickies, behind whiteboards, and under our beds. We studied the symptoms and tried tracing them back to their origin. In a nutshell, when talking about the causes of Zombie Scrum, the discussion usually ends with the questions “What are the reasons people have for using the Scrum Framework in the first place? What are they hoping to get out of it?” One persistent theme is that Zombie Scrum thrives in environments where people respond to these questions with an empty stare.
Recovering from Zombie Scrum starts with understanding the purpose of the Scrum Framework. When you understand that zombies are driven by an appetite for fresh brains, you can make the entirely sensible decision to keep as far away from them as you can. But avoiding Zombie Scrum doesn’t end with understanding the purpose of the Scrum Framework. From there follows the hard work of removing the impediments that get in the way of delivering value to stakeholders sooner. When you don’t know what you’re aiming for, it’s hard to effectively cure Zombie Scrum. Understanding the purpose also makes clear how the various experiments and interventions in this book are connected.
In this chapter, we explore the purpose of the Scrum Framework and how its elements work together to make that possible. For a more thorough refresher of the entire Scrum Framework, you can refer to zombiescrum.org/scrumframework.
“Time to hit the books, recruit! Our calculations show that you have a 0% chance of being successful when you don’t know what you’re dealing with. Cramming this information into your brain will help keep you from becoming a zombie snack.”
It’s All about Complex Adaptive Problems
What reasons do people have for adopting Scrum? The Scrum Framework is part of something called agile software development. And that’s often where the confusion starts. In our work, we like to ask people to find synonyms for the word agile. When you use a thesaurus, you find alternatives like flexible, adaptive, and nimble. These are fantastic qualities to have in environments of increased uncertainty. Scrum is designed to help you learn quickly and make adjustments based on that learning.
But does Scrum work everywhere and every time? The definition that the official Scrum Guide offers already sends us in the right direction:
Scrum (n): A framework within which people can address complex adaptive problems, while productively and creatively delivering products of the highest possible value.1
1 Sutherland, J. K., and K. Schwaber. 2017. The Scrum Guide. Retrieved on May 26, 2020, from https://www.scrumguides.org.
The key to understanding the purpose of Scrum lies in the words “complex adaptive problems.” This tiny and hard-to-miss sentence in the Scrum Guide is a rabbit hole into a different approach to a specific class of problems. Let’s break this down a bit further.
Problems
What do we mean when we talk about “problems”? It might seem trivial to even ask this question, but understanding what a problem is and isn’t, is a good start to discover the purpose of the Scrum Framework.
The English word problem originates from Ancient Greek, where it means “hindrance” or “obstacle.” Problems are those obstacles that prevent us from doing or knowing something that we need. In a very real sense, they are puzzles we need to solve in order to move forward. Just like with puzzle games, some take only a bit of effort and have a well-defined successful outcome; others require more effort and don’t have a clear successful outcome.
Within the context of product development, there are many different puzzles on a variety of levels. Some problems may be about solving a certain bug, fixing a typo, or replacing an image; other problems involve finding a way to address the needs of a group of users or coming up with a scalable architecture. Most of these problems essentially break down into many smaller problems that we need to solve.
Complex, Adaptive Problems
Problems vary in their degree of complexity. This results from the number of variables—or puzzle pieces—involved and the degree to which you know what a successful outcome looks like. Just like a jigsaw puzzle, at some point, it becomes too difficult to look at all the pieces at once to see the big picture. In order to make progress, you need to shift from purely analyzing the problem to moving pieces around on the table to see how they fit.
“Complex” means that a solution to a problem cannot be found by simply analyzing and thinking about it. There are too many factors involved and the way the pieces interact is not something you can predict up front. In product development, many variables affect our success. Although some are obvious, most are not. In our work with teams, we often ask people to brainstorm the variables that they think influence their success in implementing the solution that they have in mind. Within minutes, people generate huge lists. For example:
• The understanding of what a user needs for a particular feature
• The differences in communication styles and skills
• The mandate and support within the organization
• The skill level of the team(s)
• A clear goal and/or vision to guide decision-making
• The quality, size, and knowledge of the existing code base
• The relationship with suppliers of required components
Unlike a jigsaw puzzle, these “puzzle pieces” are abstract and hard to define. And they interact in unpredictable and unexpected ways that can only be understood in hindsight. To complicate things further, many problems in product development don’t have a clear and obvious solution, and they involve many people and perspectives that change continuously. This is what makes them “complex and adaptive.” While doing the work with others, your understanding of the problem and the solution will change in unpredictable and unexpected ways, sometimes gradually and other times very rapidly. So you’ll have to develop (new) skills and find better ways to work together.
One example of this was the development of a product to manage incidents on the Dutch railways and involved one of the authors in a supporting role. Contrary to what the customer was used to, the product was developed incrementally over a period of several years by six co-located and cross-functional Scrum Teams. One major source of complexity was how this product had to interact reliably with dozens of old and new subsystems to retrieve, synchronize, and update real-time information about what was happening on and around the tracks. In some cases, lives literally depended on the accuracy of the information. Technical complexity aside, the product was used by partners ranging from logistical companies, emergency services, passenger rail services, and other utility providers. Even seemingly straightforward items from the Product Backlog often turned out to be much harder to solve than expected as performance issues surfaced, compatibility with older systems and hardware caused problems, and the teams struggled with the political realities of having many stakeholders. Not only was the development of the product as a whole a complex and adaptive problem, so was each item on the Product Backlog. But thanks to an empirical approach, the teams were able to incrementally deliver a successful product that is still in use today and has cut response time to incidents by 60 percent.
Complexity, Uncertainty, and Risk
A key attribute of complex problems is that they are inherently uncertain and unpredictable. Because both the problem and the solution require active exploration with stakeholders, and because there is no unambiguous definition of success, what is going to happen next becomes increasingly unclear as you look further into the future. Just like the weather, you will have a pretty good idea about what is going to happen tomorrow, a general sense of what will happen next week, and no clue at all about what is going to happen one month down the road. This uncertainty inherently means there are risks. The risk of going in the wrong direction, the risk of spending time and money on the wrong things, and the risk of getting lost altogether.
A knee-jerk strategy to reduce this risk is to further analyze and overthink the problem before implementing a solution. For simple problems, this approach can work. But for complex problems, more analysis is as pointless as trying to solve a 10,000-piece jigsaw puzzle by looking at the pieces and trying to fit them together only in your head.
Yet this is how many organizations approach complex problems. They create task forces to think about solutions, spend more time in the design stage or require more detailed plans. Instead of actually moving jigsaw pieces around to see how they fit, they engage in increasingly ritualistic ways to exorcise complexity. But none of these rituals actually reduces risk because the simple truth remains that complex problems are inherently uncontrollable and uncertain.
Thankfully, there is an excellent way to actually reduce the risk of complex, adaptive problems. This is where Empirical Process Control Theory and the Scrum Framework come into view.
Empiricism and Process Control Theory
We are surrounded by complex problems. Even seemingly straightforward problems turn out to be complex on closer inspection. One way to find an answer to these problems is through reasoning or using intuition. You can also rely on previous experiences. But how reliable is experience when you’ve never done something before or when the variables change all the time?
Chemical engineers have long wrestled with complex challenges too. As it turns out, even seemingly straightforward chemical processes are complex on closer inspection. How do you keep the temperature of a liquid constant? How do you heat crude oil for transport without reducing its quality? So many variables influence these processes that it requires a different approach to control them. This topic is what has come to be known as Empirical Process Control Theory.2 Instead of trying to identify all possible variables and their interactions in comprehensive models, important key variables are constantly monitored with sensors. When their values exceed certain thresholds, other variables are modified to adjust the system back to the desired state. More heat can be applied, air can be vented, water can be added or removed. Here, knowledge to drive decisions does not come from models or assumptions. Instead, it comes from a short feedback loop where adjustments are made as needed based on frequent measurements.
2 Ogunnaike, B. A., and W. H. Ray. 1994. Process Dynamics, Modeling, and Control. New York: Oxford University Press.
This way of developing knowledge from experience is called “Empiricism”. Developed as early back as ancient Greek times, it is the foundation of modern science. It contrasts with rationalism, where analysis and logical reasoning are used to arrive at knowledge. In empiricism, nothing is assumed to be true until it has been verified through observation.
Although Empirical Process Control was developed in part to control complex chemical processes in industrial plants, its principles can be applied to complex problems in other domains all the same. The Scrum Framework is one example of such an application.
Empiricism and the Scrum Framework
The Scrum Framework was developed by Ken Schwaber and Jeff Sutherland in the 1990s and first formalized in 1995 to address the inherent complexity of product and software development.3 More recently, the Scrum Framework is being applied to complex problems in a variety of domains, such as marketing, organizational change, and scientific research. The Scrum Framework is built on three pillars that allow empirical process control (see figure 4.1):
• Transparency. You gather data—such as metrics, feedback, and other experiences—to find out what is going on.
• Inspection. You inspect the progress with everyone involved and decide what that means for your ambitions.
• Adaptation. You make changes that you hope will bring you closer to your ambitions.
3 Sutherland, J. V., D. Patel, C. Casanave, G. Hollowell, and J. Miller, eds. 1997. Business Object Design and Implementation: OOPSLA ’95 Workshop Proceedings. The University of Michigan. ISBN: 978-3540760962.
Figure 4.1 The short cycles of creating transparency, inspecting the outcomes, and adapting what else is needed
This cycle repeats as often as necessary to catch deviations, unexpected discoveries, and potential opportunities that emerge as the work is done. The process happens not once a year or when the project is completed, but continuously on a daily, weekly, or monthly basis. Rather than basing our decisions on risky assumptions about potential futures that will probably never unfold, we’re instead making decisions based on the signals we’ve detected up to this point. This is empiricism. And you’ll discover later in this chapter that everything in the Scrum Framework is designed around these pillars.
What the Scrum Framework Makes Possible
The empirical approach that the Scrum Framework offers becomes tremendously useful when you accept that you don’t know everything and can’t control every variable. Because of that, your understanding of what is needed changes. You have to accept that mistakes and new insights will emerge that you never considered initially. Instead of making a precise plan up front and then sticking to it no matter what, you have to treat ideas as assumptions or hypotheses that you validate with the Scrum Framework.
The Scrum Framework allows you to learn whether you’re off track and need to make adjustments much sooner than when you’re simply following a plan. Instead of going all in on one solution, you’re now able to tackle the biggest problem you’re facing first.
This is especially important when you operate in an uncertain, changing environment. Reasonable assumptions you have at the beginning of your work may fly out the window as you’re developing your product. Instead of catastrophic failure at the end of a long project, an empirical approach reduces unexpected changes to minor speed bumps that require you to correct course a bit.
If anything, the Scrum Framework helps to reduce the risk of the inherent unpredictability and uncertainty of complex, adaptive problems. It allows you to continuously verify that you’re still moving towards solving the problem you set out to solve. Even better is that you now have a process that actively encourages the discovery of better ideas and includes them in shaping your next steps. Now uncertainty becomes an asset because of all the underlying possibilities within it.
Scrum: An Evolving Set of Minimal Boundaries to Work Empirically
When you read the Scrum Guide, or our description of the Scrum Framework,4 you may notice that the Scrum Framework leaves many things open. For example, how do you define Sprint Goals? How do you create cross-functional teams? What are practices that help Product Owners or Scrum Masters be successful? Coming to the Scrum Framework with a fresh eye may be a frustrating experience, as people looking for a complete methodology understandably wonder, “But how do I do this?”
4 Download the PDF from https://zombiescrum.org/scrumframework.
The Scrum Framework is incomplete on purpose. It is best understood as a minimal set of boundaries for working empirically. It describes only what you need to do, not how to do it. The Scrum Guide makes no mention of specific practices such as test-driven development, story points, or user stories. Every team, product, and organization is different. This complexity means that there is no silver bullet or one-size-fits-all solution. Instead, the Scrum Framework encourages teams to discover their own local solutions and ways of doing things within each team’s boundaries. There are many potential sources for learning what might work, from simply trying different things to getting inspiration from blog posts, podcasts, and meetups.
The Scrum Framework isn’t static either. It has changed over time. Since its first formal description in 1995, the collective insights and experiences of teams working with Scrum have resulted in many small and large adaptations. A general trend is that the Scrum Framework is increasingly applied in domains outside product and software development. This is reflected in the removal of specific practices—such as burndown charts—from the framework. Wording has changed to emphasize the intention over the implementation. Newer versions increasingly emphasize the importance of Sprint Goals and values to drive decision-making in complex environments. The Scrum Framework itself is subject to its own process of transparency, inspection, and adaptation.
Zombie Scrum and the Efficiency Mindset
Where does Zombie Scrum connect to all this? One clear theme we’ve found is that people use the Scrum Framework for the wrong reasons. When you ask people in a Zombie Scrum organization what they are hoping to get out of Scrum, you’ll hear things like “more speed,” “more brains,” “more output,” and “more efficiency.” That’s very different from the actual meaning of the word agile. It’s also very different from what the Scrum Framework is designed for. Where does this contradiction come from?
The traditional way of managing organizations and product development is designed to achieve the opposite of agility. This mental model is often called the efficiency mindset. A full history of the efficiency mindset is beyond the scope of this book, but Gareth Morgan’s work offers a good introduction.5 Suffice to say that its aim is to reduce uncertainty as much as possible, increase predictability, and drive efficiency. This mindset usually manifests in detailed plans for upcoming work, the standardization of work through protocols and procedures, a high degree of task specialization, and measuring efficiency (such as units of work per day, resource utilization, or number of errors). This mindset can certainly work in environments where work is fairly repetitive and simple, such as assembly lines or certain administrative work. But it certainly doesn’t work in environments where people deal with complex, adaptive problems that are inherently unpredictable and uncertain.
5 Morgan, G. 2006. Images of Organization. Sage Publications. ISBN: 1412939798.
And yet this mental model is so ingrained that it’s effectively invisible. It completely shapes the way we design organizations, structure interactions, and build our culture. When you look at the Scrum Framework from this perspective, it makes sense that people try to understand the framework in terms of how it impacts efficiency, speed, and output—only to be disappointed when it doesn’t seem to do that.
In a very broad sense, the Scrum Framework is more concerned with being effective than being efficient. Where efficiency is about getting as much work done as possible (the output), effectiveness is about the value and usefulness of that work (the outcome). Although it is entirely possible that efficiency improves with the Scrum Framework, it is neither a promise nor a goal in itself.
In environments where Zombie Scrum is going on, the efficiency mindset is so strong that people see only the structural elements of the Scrum Framework: the roles, events, and artifacts. They don’t see nor appreciate the value of the empirical process underneath (see Figure 4.2). And that is why Zombie Scrum only looks like Scrum, but without the beating heart of empiricism.
Figure 4.2 Looking at the wrong things? Zombie Scrum goes hand in hand with a strong focus on performance and how much work is done. But are customers happy? Is value delivered?
What about Simple Problems?
When the Scrum Framework is designed for complex and adaptive problems, what does that mean for situations where you are dealing with simpler problems? What if you’re in doubt about whether or not you’re confronting a complex problem in the first place?
First, the people doing the work often have the best chance of detecting complexity. What may seem incredibly straightforward for a stakeholder may be very hard for developers. One of the authors once encountered a stakeholder who boldly stated that building a web shop involved nothing more than putting a USB stick into a laptop. Obviously, this stakeholder benefited from this belief, as he hoped it would keep the price of development down. But we all have examples of situations where someone who is not doing the work claims that “it can’t be that difficult.” If anything, complexity should be judged by the people doing the work.
Nevertheless, even the people doing the work can easily fool themselves. Complex problems are deceptive in that their complexity is often not obvious at first glance. It’s only when you start solving them that you discover there is a lot more beneath the surface. Most developers know what this is like: they start with a seemingly small change, only to discover that this small change affects many other components, and cascades into unexpected issues. What started out as a seemingly simple problem turned out to be a complex one instead.
A third consideration is that complexity doesn’t have to spring from technical matters alone. Although changing the text on a button on a website may be easy, complexity also emerges when many groups of stakeholders have to be involved. As the number of people in the mix increases, complexity tends to increase along with it.
Finally there is the question of scale. Even the most complex problem can be broken down into small tasks that are in themselves straightforward and simple. In a sense, this is exactly what we’re doing in the Scrum Framework: a large problem is decomposed into a series of smaller problems that each fits into a Sprint. Those smaller problems are in turn broken down into even smaller problems that are represented by the items on the Sprint Backlog. Sometimes all we see are the simpler tasks on the Sprint Backlog. Using this to conclude that the problem isn’t complex ignores the bigger picture.
When we take these considerations into account, we firmly believe that the majority of the problems we face in the modern workplace are complex and benefit from empiricism in one form or another. The few exceptions generally involve work that consists of repetitive tasks where each can be successfully completed without coordination with others. When in doubt, you’re better off to assume complexity and rely on an empirical approach such as Scrum, Kanban, DevOps, or Extreme Programming. If the problem really turns out to be simple, you will quickly notice that an empirical approach doesn’t yield new insights or useful adaptations. In this case, the decision that an empirical approach is not useful is made empirically too. And it helps to avoid the risk of assuming that something is simple, only to discover that it isn’t and having to rethink your entire approach and the expectations you’ve created.
More detailed analyses on how to distinguish simple from complex problems, as well as recommended approaches, can be found in work by Ralph Stacey6 and Cynthia Kurtz and Dave Snowden.7
6 Stacey, R. 1996. Complexity and Creativity in Organizations. ISBN: 978-1881052890.
7 Kurtz, C., and D. J. Snowden. 2003. “The New Dynamics of Strategy: Sense-making in a Complex and Complicated World.” IBM Systems Journal 42, no. 3.
Now What?
Zombie Scrum happens when teams lose focus on, or don’t understand, the purpose of the Scrum Framework. That is why we used this chapter to explain how the Scrum Framework exists to help teams navigate the inherent risk of complex problems. It is not a detailed methodology that you can execute without thinking. Instead, the Scrum Framework offers a minimal set of boundaries that allow teams to work empirically on complex problems of any kind. In its simplest form, it encourages teams to work collaboratively in small steps to solve complex problems together with those who have a stake in the problem. Every step is used to learn about what else is needed, to validate assumptions, and to make decisions about the next steps.
It is true that the Scrum Framework is easy to learn. But it is hard to master. Every journey with the Scrum Framework starts somewhere. Whatever your starting point is, the best way to learn how to work with Scrum is to do it. The iterative and incremental nature of the Scrum Framework is a great vehicle for learning and improvement when you keep its purpose in mind. Although the journey may be hard, even seemingly impossible at times, improvement will happen over time. And thankfully there is a huge and passionate community of people working with Scrum around the globe, ready to help you. And there’s this book, of course. In the coming chapters, we explore the symptoms and causes of Zombie Scrum in more detail and offer practical experiments to recover.