Release on Demand

Develop on Cadence. Release on Demand.

—A SAFe mantra

Release on Demand is the process by which Features deployed into production are released incrementally or immediately to Customers based on market demand. It is the fourth and last element in the four-part Continuous Delivery Pipeline, shown in Figure 1.

Figure 1. The final element of the continuous delivery pipeline: release on demand

The three processes that precede release on demand in Figure 1 help ensure that the value is ready to be deployed in the production environment. But since tangible development value occurs only when end users are operating the Solution in their environment, delivery value at the right time is critical for the enterprise to gain the real benefits of agility. The decision to release is a key economic driver that requires careful consideration. While deployed features may be immediately available to all end users in some situations, more often the release is a decoupled on-demand activity, occurring for specific users, timed for when they need it.

Details

The ability to release on demand is a critical aspect of the continuous delivery pipeline, which raises three questions that allow Agile Release Trains (ARTs) to embrace Principle #3, Assume variability and preserve options:

1. When should we release?

2. Which elements of the system should be released?

3. Which end-users should receive the release?

Having all three options accelerates the delivery process and provides flexibility in delivering value. Further, release on demand ‘closes the loop’ by evaluating the benefit hypothesis proposed during the Continuous Exploration stage of the pipeline. Releasing provides the data needed to decide whether further exploration of an idea is warranted, or if new functionality should be added, or if perhaps a different approach is warranted.

Practicing release on demand requires the organization to develop the following capabilities:

• Decoupling the release from the development cadence

• Decoupling the release elements from the solution

• Designing the solution to support incremental releases

The following sections describe each of these capabilities.

Decouple the Release from the Development Cadence

Develop on cadence is the other half of a strategy that allows ARTs and the Solution Trains to operate in a predictable pattern and synchronize the efforts of multiple development teams. When it comes to releasing value, though, a different set of rules may apply.

Once we have a reliable stream of value through the Continuous Deployment process, the next consideration is when and how to release. The release strategy is decoupled from the development cadence, as shown in Figure 2.

The figure shows two rightward arrows labeled “release on demand” and “develop on cadence” at the top and bottom, respectively. Five vertical bars labeled “PI” are shown in between the arrows. Between each PI four iterations are shown. Five rectangular boxes above the iterations are shown. An upward arrow from each box reads, customer preview, major release, subsystem release, major release, and a new feature.

Figure 2. Decoupling development concerns from release concerns

Several release strategies may apply depending on the context and situation:

• Releasing on the Program Increment (PI) cadence – In the simplest case, the enterprise releases on the PI boundaries. With this strategy, PI planning, releasing, and Inspect and Adapt will have predictable calendar dates. Also, the Innovation and Planning Iterations can be timed, designed, and organized to support the more extensive release activities. The IP iterations may include final, Verification and Validation (V&V), User Accepted Testing (UAT), and training and release documentation.

• Releasing less frequently – In many cases, releasing on a fast PI cadence may not be possible or even desirable. For example, in some enterprise settings, deployed systems constitute critical infrastructure for an operating environment. Even if the customer would like to have the new software, the service level and license agreements may rule out this kind of release. Moreover, the organization must take into account the overhead and disruption associated with deployment. In other cases, the timeline for building complex systems that include both software and hardware (e.g., mobile phones or geophysical mapping satellites) is driven by those systems’ hardware components (displays, chipsets, and the like), which typically have longer lead times. The new hardware must be available first, which means releasing early and incrementally may not be an option. In these cases, releasing on a PI cadence may not be practical, and the planning and releasing activities are entirely decoupled.

• Releasing more frequently – For many organizations, the goal is to release as often as possible—hourly, daily, weekly, and so on. Achieving frequent releases requires DevOps capabilities, an efficient continuous delivery pipeline, and architecture that supports incremental delivery. Even then, the periodic planning function still provides the cadence, synchronization, and alignment the enterprise needs to manage variability and limit deviations from expectations.

• Release on demand – For organizations building complex solutions (e.g., systems of systems), the preceding cases are probably overly simplistic. Big systems often contain different types of components and subsystems, each of which may leverage its own release model. In that case, the most general model is this: Release whatever you want and whenever it makes sense, within an appropriate governance and business model.

Decouple Release Elements from the Solution

Applying different release frequencies raises another question: Is a release a monolithic, all-or-nothing process? If so, the release strategy would be limited. Fortunately, that’s not the case. In fact, even simple solutions will have multiple release elements, each operating with different release strategies, as Figure 3 illustrates.

The figure shows a rectangle labeled “solution” consisting of three rectangles inside it. To the right, a horizontal cylinder comprising of streamlet 1, streamlet 2, streamlet 3, and streamlet 4 are shown. An arrow labeled “end-user functionality (released every 2 weeks)” from one of the inner rectangles points to streamlet 1 followed by seven rectangular boxes. An arrow labeled “security updates (released on demand)” from the other inner rectangle points to streamlet 2 followed by three rectangular boxes. An arrow labeled “back-office functionality (released every month)” from the third inner rectangle points to streamlet 3 followed by four rectangular boxes. An arrow labeled “entire solution (a major release every quarter)” from the outer rectangle points to streamlet 4 followed by two rectangular boxes. Text above the solution, cylinder, and rectangular boxes read, feature toggles, canary release, and dark launches. Three vertical bars labeled “PI” are shown below the arrows. In between the bars, four iterations are shown. Three circular arrows intersecting are shown before the green rectangle: a clockwise arrow labeled “continuous exploration,” a counter-clockwise arrow labeled “continuous integration,” and a clockwise arrow labeled “continuous deployment.”

Figure 3. Decoupling release elements from the solution

For example, the SAFe website has multiple release cycles:

• Its authors can make a fix to a deployed version or address security at any time (ad hoc, but expedited).

• Its authors can update any article at any time and simply notify readers via blog post (high frequency).

• Its authors can add new articles to the Advanced Topics section whenever significant new content is available (medium frequency).

• Significant revisions to the framework are made only periodically, based on new content, new ways to render the Big Picture, and most importantly, when the market is ready for a new release (low frequency).

We call these separate flows value streamlets, as they represent a full, end-to-end flow of value within a Value Stream. Each streamlet (or value stream segment) can and should be managed to deliver value according to its own needs and pace. Identifying streamlets is critical to enable release on demand, as they allow the different elements of the solution to be released independently in a separate cadence.

Architect the Solution for Incremental Release

Achieving the different release strategies requires solutions to be designed for component (or service-based) deployability, releasability, and fast recovery. Intentional design of these attributes requires the collaboration of System Architects, Agile Teams, and deployment operations. The goal is a quick and flawless delivery process, ideally requiring little manual effort.

The following capabilities allow for a more flexible release process:

• Feature toggles – Feature ‘toggles or switches’ enable deployment of features into production without making them visible to the end user. They help avoid the need for multiple code branches, allowing developers to deploy new features to production but activate them only when the enterprise is ready to consume them. After the feature has been released and deemed stable, the toggles are removed to avoid technical debt.

• Canary releases – This technique reduces the risk associated with introducing a release in production, by slowly rolling out the change first to a small subset of users, and only then rolling it out to the entire user base. Canary releases allow selective production testing and feature validation without impacting everyone.

• Dark launches – In some situations, it’s important to test new functionality in a production setting before making it available to customers, especially when there is high uncertainty on how well the system will meet its Nonfunctional Requirements (NFRs) under a production load.

Implementing these capabilities may require the organization to upgrade its infrastructure and architecture, including moving to standard technology stacks, decoupling monolithic applications with micro-services, ensuring data preparation and normalization, engaging with third-party APIs and data exchange, and implementing logging and reporting tools.

Close the Loop on the Feature Hypothesis

Finally, after the feature is released, it’s time to evaluate the benefit hypothesis. Were the intended outcomes achieved? For example, should a canary release be extended to more customers? Should the organization turn off the feature toggles? The feature is considered done once we have an understanding of the production results. However, it may be necessary to add new backlog items to extend the functionality (persevere), or pivot (remove the feature) to find a different solution.

Building the Release

Building large-scale systems that are ready for release and fit for use requires an incremental development approach, as Figure 4 illustrates. The four types of increments are described next.

Four rectangular boxes of size varying from small to big are shown placed one inside the other. The boxes from small to big are labeled as follows: team increment, system increment, solution increment, and releasable solution. An arrow from releasable solution reads release: additional activities, including system validation, documentation as required by the business. An arrow from solution increment reads, solution increment: ARTs assets integrated, tested, demonstrated at solution demo at least every PI. An arrow from system increment read, system increment: teams backlog items completed, integrated, demonstrated at system demo at least every iteration. An arrow from team increment reads, team increment: team finished, iteration backlog stories, demonstrated at team demo at least every iteration.

Figure 4. Building a releasable solution

Team Increment

The first step in this process is that each Agile team ensures that it produces a working increment for the stories, features, and components the team is responsible for by doing the following:

• Completing user stories from the Team Backlog, while assuring that each meets its local (story) Definition of Done (DoD)

• Applying Continuous Integration practices with test automation to monitor and confirm progress

System Increment

Every two weeks, teams build a system increment—that is, an integrated stack of new functionality, representing all the backlog items completed by the ART during the current and all previous iterations. During the System Demo, stakeholders review the system increment and provide feedback. In this way, new features are added incrementally to the system, usually a few per iteration.

Solution Increment

When developing large solutions, ARTs typically contribute to only a part of the solution, whether it’s a set of subsystems, some solution capabilities, or a mix of the two. When integrated, verified, and validated, the solution increment must satisfy both the functional and Nonfunctional Requirements. Such a solution increment is the subject of the all-important Solution Demo, which brings all the system increments together into a working system.

Release Increment

Building solutions incrementally affects the release increment as a whole. As discussed earlier, decoupling solution elements allows for smaller, independent releases of the solution. These items might be at the team, system, or solution levels, so the same logic should be applied across the entire integration stack.

Some additional activities might be required when releasing increments—namely, final, V&V, release notes, UAT, documentation, and training materials, as well as marketing activities. As much as possible, these activities are part of the DoD of previous increments. Some, however, will remain as release activities.

A Scaled Definition of Done

The continuous development of incremental system functionality requires a scaled DoD. An example is shown in Table 1.

The column headers read, team increment, system increment, solution increment, and release. A rectangular block is shown above the first column. Two rectangular blocks placed one inside the other is shown above the second column. Three rectangular blocks placed one inside the other is shown above the third column. Four rectangular blocks, placed one inside the other is shown above the fourth column. The first column reads, stories satisfy acceptance criteria; acceptance tests passed (automated where practical); unit and component tests coded, passed, and included in the BVT; cumulative unit tests passed; assets are under version control; engineering standards followed; NFRs met; no must-fix defects; and stones accepted by Product Owner. The second column reads, stories completed by all teams in the ART and integrated; completed features meet acceptance criteria; NFRs met; no must-fix defects; verification and validation of key scenarios; included in build definition and deployment process; increment demonstrated, feedback achieved; and accepted by Product Management. The third column reads, capabilities completed by all trains and meet acceptance criteria; deployed/installed in the staging environment; NFRs met; system end-to-end integration, verification, and validation done; no must-fix defects; included in build definition and deployment/transition process; documentation updated; solution demonstrated, feedback achieved; and accepted by Solution Management. The fourth column reads, all capabilities done and meet acceptance criteria; end-to-end integration and solution V&V done; regression testing done; NFRs met; no must-fix defects; release documentation complete; all standards met; and approved by Solution and Release Management.

Table 1. Example of a scalable DoD

Release Management

Release management is the process of planning, managing, and governing solution releases, which helps guide the value stream toward the business goals. In some enterprises, especially those that must meet significant regulatory and compliance criteria, a centralized, portfolio-level team or function is present that assures releases meet all the relevant business criteria. In other circumstances, ART and Solution Train leadership and stakeholders from development operations, quality, sales, and other stakeholders should own some of the release management and governance responsibilities.

In either case, the release management function facilitates the activities needed to help internal and external stakeholders receive and deploy the new solution. It also ensures that the most critical governance quality elements are appropriately addressed before deployment—particularly internal and external security, regulatory, and other compliance guidelines.

Release planning is part of the PI planning process. But that’s the easy part; the hard part is coordinating the implementation of all the capabilities and features over multiple iterations within a release. This is especially true when new issues, roadblocks, dependencies, and gaps in Vision and backlogs are uncovered. Due to these challenges, the scope of each release must be continually managed, revalidated, and communicated. Primary considerations include the following:

• Ensuring that the organization’s release governance is understood and adhered to

• Communicating release status to external stakeholders

• Ensuring that an appropriate deployment plan is in place

• Coordinating with marketing and with Product and Solution Management on internal and external communications

• Validating that the solution meets relevant solution quality and Compliance criteria

• Participating in Inspect and Adapt (I&A) workshops to improve the release process, value stream productivity, and solution quality

• Providing final authorization for the release

• Acting as a liaison with Lean Portfolio Management (LPM), as appropriate

• Participating in and overseeing the final release activities

Many enterprises hold release management meetings weekly to address the following questions:

• Is the vision still understood, and are the trains and teams aligned for that purpose?

• Does everyone understand what they are building, and is their understanding aligned with the purpose of the Value Stream and current Strategic Themes?

• Are the trains tracking to the scheduled release dates?

• Is the appropriate quality built into the solution?

• Which impediments must be addressed to facilitate progress?

This weekly meeting provides senior management with regular visibility into the release progress. It’s also the place to approve any scope, timing, people, or resource adjustments necessary to assure the release. In a more continuous delivery environment, the participants closely monitor the release section of the Program Kanban, making sure items are released when needed, to the right audiences; managing dark and canary releases; verifying that hypotheses are evaluated; and ensuring that feature toggles are removed after production verification.

LEARN MORE

[1] Kim, Gene, Jez Humble, Patrick Debois, and John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press, 2016.