Agile has been around since 2001, but it still isn’t well understood in the industry. Many organizations do one or a few of the easier Agile practices like stand-up meetings and two-week sprints, and claim they’re doing Agile. Most Scrum teams know there’s a Product Owner and work in fixed time horizons called time boxes. These are important practices, but are valuable only if adopted properly, fully, and in conjunction with other important practices.
It’s not about Waterfall versus Agile. It’s common for many of the Waterfall processes to seep into the Agile practices. For example, when requirements gathering becomes so complex that requirements have to be written down to later be read and interpreted, it can introduce major inefficiencies and bugs.
And keeping a separate QA process where developers pass software to testers for verification is expensive and inefficient. Retaining these practices and adopting two-week sprints and stand-up meetings won’t get you much improvement.
I prefer to focus on practices that address the specific forces of developing software rather than on labels like Waterfall or Agile. This book is not saying that Waterfall is bad and Agile is good. That’s far too simplistic to address the core issues.
In my experience, one of the main benefits of developing software in an Agile environment is that it supports making development a discovery process; the team constantly receives feedback and learns from it. Agile requirements—or as Agile calls them, stories—do not by themselves replace the sort of lengthy, detailed specifications that Waterfall demands. Stories are supposed to spark conversations—requiring and encouraging meaningful interaction between software developers and the Product Owner. It’s from this interaction that developers learn enough about what’s needed to build it.
Just removing specifications without replacing them with conversations between developers and the Product Owner is not the intent of Agile. And if all the Product Owner is doing is taking specifications from the customer and creating detailed requirements, then maintaining documentation along the way—that’s just Waterfall with a Product Owner. It’s not Agile.
We need to look deeper to find the hidden essence of Agile, which is not just having a Product Owner and getting rid of specifications but really changing the conversation to the what and the why instead of the how.
For example, a central purpose behind the practice Build in Small Batches is to take tasks from start to finish as quickly as possible, and smaller tasks can be taken to completion quicker. Many teams do iterative development, but then their work waits in a queue to be validated by a QA team right before release. They take their work to 99% completion, but that’s not good enough. Their code harbors an unknown amount of risk that is revealed only after integration and testing.
When people understand that the purpose of building in small batches is to take tasks to completion as quickly as possible and to limit work-in-progress, they apply the practices more advantageously and begin to see bigger benefits.
Likewise, time boxing can be misunderstood and misapplied. The practice of breaking big problems down into smaller problems using time boxes in Extreme Programming is called iterations, and the term in Scrum is sprints. But I’m not a big fan of either of those terms. They tend to give the wrong connotation.
Agile and Scrum are not about rushing through work but working in small chunks. Agile is really all about scope boxing—limiting the scope of what we’re working on—and we really only time box in order to get people familiar with scope boxing.
Time boxing is simply saying, “We’re going to work on this for a fixed amount of time,” and usually that’s a very short horizon—in Scrum they say one to four weeks—and typically the cadence is every two weeks for an iteration or sprint.
The key, though, is that we should be trying to think smaller in terms of scope or units of work, rather than units of time. We break down big tasks into smaller units of work that still produce an observable result. And the smaller the unit of work the better—smaller tasks are easier to estimate, implement, and verify.
Scrum caught on as a methodology to manage software development and all but forgot about supporting the technical practices of Extreme Programming. Scrum encourages teams to self-organize, which sounds like a great idea, but just telling developers to self-organize doesn’t provide enough guidance to have them adopt practices that help them focus on code quality and write maintainable software.
Just having to go back to code written a few weeks ago and add features can be enough to drive many developers to start writing more maintainable code. But when the technical practices are ignored, improperly adopted, or just generally misunderstood, we end up with teams saying “We do Scrum” while they’re really still operating in a requirements-heavy, test-last Waterfall environment. They’re just writing smaller batches of code, but it’s still full of dependencies that make it hard to work with and bugs that aren’t discovered until much later.
I’ve seen Scrum teams improve their immediate productivity only to find, four or five years into a project, that they’ve accumulated so much technical debt and their code is so poor that it’s nearly impossible to work with. Ultimately, they end up having to address their poor development practices and dig themselves out of the hole they created with poor-quality code before they can become productive again.
I’ve seen teams adopt Scrum and Extreme Programming but utterly fail. I’ve seen teams do Waterfall and succeed. Every approach has benefits and drawbacks. We must understand the purpose of every practice to use them correctly.
We’ve seen how the forces of software engineering are very different from anything in our previous experience. Even electrical engineering and mechanical engineering are based on physics. But software does not obey physical laws, and we sometimes have a difficult time relating to it and really understanding it in a detailed way.
Nearly everything in the physical world is fault-tolerant. Living and non-living systems have enormous resiliency. But software is by far the most fragile thing in our world. One incorrect bit can cause a catastrophic systems failure. Because of this single fact, we must build software in a verifiable way.
To many, software development is counterintuitive. Quality control standards that revolutionized manufacturing utterly fail when applied to software projects. Much of what we learned in the Industrial Revolution is meaningless when it comes to software. It’s an entirely different animal.