CHAPTER 9

Different Delivery Models

Gentlemen, we are going to relentlessly chase perfection, knowing full well we will not catch it, because nothing is perfect. But we are going to relentlessly chase it, because in the process, we will catch excellence.

—Vince Lombardi, as quoted in Game of My Life by Chuck Carlson

In this chapter, I will describe the three different IT delivery models that are currently being leveraged for successful IT delivery. I will also describe the different capabilities required to make them work. But as I have mentioned a few times already, the challenge goes beyond just the technical uplift to master those capabilities; it depends on the organization to make the appropriate changes across the organization to support the delivery model.

Overview of Delivery Models

I have seen three models that are actively being used or targeted within large organizations to deal with legacy technologies and modern digital technologies at the same time:

1. Continuous delivery: This is the kind of delivery that was described in Jez Humble and David Farley’s book on continuous delivery, which allows you to automatically deploy applications into all the environments from development to production and automatically test them. It is based on persistent environments for deployment.1
2. Cloud-enabled delivery: This is the delivery model that Netflix made popular, which creates a new environment each time and destroys the previous one once the new environment has proven to be working. This is effectively a zero-downtime deployment.2
3. Container-enabled delivery: This is the delivery model that has become popular with the rise of Docker as a container technology that supports a microservice-based architecture.

There is a fourth delivery model evolving at the moment, based on serverless technologies like Amazon Lambda. At the time I am writing this book, I have not worked with clients to define delivery models for serverless technologies and have not seen a formulated delivery model for them. Perhaps in the next version of this book, I can extend the chapter to include this fourth delivery model.

Within your organization, you will likely have multispeed delivery concerns, which means you will use a mix of these delivery models. When I speak about the transition impacts toward each of these models, remember that this will not be an everyone-at-once approach but will be rather gradual as you move applications and technologies to these delivery models.

Delivery Model A: Continuous Delivery

This model is probably the most well known and has been around for a while, though many companies still struggle to implement it effectively. Continuous delivery means that you can potentially deploy into production with every build, as all the required steps are automated. The word “potentially” is to be read the same way it is used in the Agile community, which means the importance is on the ability to do so, not on actually doing it. You might still choose not to deploy automatically into production—for example, to allow for manual testing or hardening later on. You can leverage this delivery model for both cloud or on-premises environments.

The most common environment pattern into which to deploy is that of persistent environments (e.g., environments that have more than one software drop being deployed). This is often a necessity when working with legacy applications that require a very specific environment setup, so this model is very well suited. The benefits of moving to this model from a more manual, traditional delivery model are a significant improvement of delivery speed, a removal of risk out of your delivery by removing manual steps and increasing the frequency of inspection and feedback, and a reduction of unhelpful noise from the delivery process by increasing transparency across the delivery life cycle.

Figure 9.1: Model A—Continuous delivery: Continuous delivery automates delivery to persistent environments

Description of Capabilities

Continuous delivery is supported by four different capabilities that you need to master. I will only provide a short overview to describe the basics, as there is a lot of material already available to go deeper if you need to.

1. Creating the application: This is one of the core capabilities that enables everything else in the process. This capability covers source-code management; the developer side-quality process (static code analysis, peer reviews, unit testing); the developer work-management process (being able to trace all changes to the functional request associated with it); the compilation, build, and packaging process; and the package management. Mastering this capability is crucial for everything else that comes. While there are successful patterns to leverage here for certain technologies (e.g., you will be able to Google and download Jenkins settings for the building of Java applications), the capability will continue to evolve and will be different in the context of each of the technologies you use.
       Ideally, you get to continuous integration (CI), where your application is built with each check-in as a pattern that you can use across all your applications and technology; but often, you will encounter some technologies for which this is just not feasible in an economic fashion. For example, when working with Siebel, our compilation time was over two hours, which is too long for CI. However, automating this capability is probably the easiest of them all. The frequency might differ, but there should be very minimal manual effort involved to create a software package from the code in the software configuration management (SCM) system. I have automated this for Siebel, mainframe, and many other technologies. It wasn’t always easy but was possible.
2. Deploying the application: Deploying is already a bit more complicated. Overall, it means we are picking up the software package from the package management system and are deploying it into an existing environment. This is also very suitable for full automation. To do that, there are few things to consider.
   • Knowing the environment: To deploy successfully, you need to know where to deploy the application. It is likely that you have a topology of environments with differences across levels (e.g., multitenancy in lower levels and redundancy of components closer to production), so you need to know on which server which components need to be deployed. You also need to know the environment-specific settings (like IP addresses, server names, connection values) at the time of deployment; and for incremental deployments, you need to know the last version that was deployed.
   • Keeping software packages environment independent: Because you never know where the software package will be deployed when you create the package, all environment-specific settings need to be abstracted away. You can do this through a configuration file or by leveraging variables that either get resolved at deployment time or at execution time.
   • Full versus incremental deployments: A full deployment will completely replace the application in the environment, which means you don’t need to worry about anything. You first delete everything associated with the previous version (after making a backup, of course, in case the new deployment fails) and then deploy the new version into the environment. This full deployment is a lot easier to automate, as it will always follow the same process.
         Incremental deployments are faster but require more complex processes to optimize, which is why organizations often start with full deployments unless the technology does not allow for it (e.g., structure upgrades for transactional tables are always incremental). For incremental deployments, knowing the sequence of incremental packages and which version is in the environment is critical to identigying the right delta set (the files that require changing). It is good practice to automatically validate that the environment is in the expected state before deploying, as the risk of something failing is much higher with incremental deployments. There is also a larger risk of configuration drift from manual changes or failed deployments because the environment is not being cleaned up with each deployment.
3. Testing the application: We want to automate as much of the testing process as is feasible and account for the scope of testing, which differs between environments. All the different levels of testing fall into this capability: application test, integration test, performance test, security test, operational readiness test, and anything else that can be automated. Strictly speaking, all the manual testing and how-to-manage manual testing also falls into this capability.
4. Visualizing the delivery process: I believe that you cannot improve what you don’t see. The overall delivery model is not something you implement once and that’s it; you will have to keep tuning it and improving it. In my experience, the first implementation takes three to six months; then it still changes a lot for the next six to twelve months as you improve. To do this right, you need to have a way of visualizing the end-to-end process and to measure the activities for accuracy and speed. In the past, this was done with text files and Excel sheets like so many things in IT, but new visualization tooling and open-source solutions have allowed this capability to become easy to implement and sexy to use. Capital One went so far as to open-source their internal solution for a DevOps dashboard, which has been adopted by other organizations successfully to manage their DevOps adoption.3 Out of all the capabilities in this delivery model, this is not the most difficult one but often the most undervalued one. Way too many organizations don’t spend enough time and energy on this capability.

Transition Concerns and Organizational Impact

As you transition to continuous delivery, there are a few things you want to consider. First of all, configuration management is crucial; without that, you really can’t do anything else. Configuration management allows you to operate at speed. All code (including the tests and the automation code) needs to be in a configuration management system so that it can be accessed and used reliably. The transition to this model requires that your operations and infrastructure teams work closely with the platform team to implement abstract environment configuration (a practice that places variables instead of concrete values in configuration files that are replaced at deployment time, when the true values are known). And you will need to have the right environment access for your automation. This will feel like a loss of control to those operations teams, but if you manage this process carefully by involving all groups in the necessary change management, the transition will go much more smoothly.

Change management is also crucial for the transition to this delivery model and all the others. I have been part of several projects to implement new delivery models, and initially, I underestimated the change management efforts (training people, motivating the change in the organization, communicating about the changes and benefits, updating process and role descriptions). After all, if we build a great solution, everyone will jump on the bandwagon, right? Not at all, it turns out. After I noticed this challenge in my first couple of projects, I started to factor this in and staff a dedicated change-management person for my subsequent projects. It turns out that change management is absolutely required and helps everyone on the team. Developers are not that interested in creating training material or process documentation, and the change-management people know how to generate support material that people actually want to use. I think you can come up with a cost-and-effort estimate for this and then double that estimate; and you will probably still look back at the end and think you should have done more.

The organizational changes for the quality organization mentioned in chapter 7 will have to be in place for this model, as you will otherwise have too much friction between the delivery teams who are highly optimized for speed and the separate testing organization.

One last thing to consider is the infrastructure for the tooling platform. Very often, this does not get treated like a production system. But think about it: when your production is down with a defect and your SCM and automation tooling is also down, you are in serious trouble. You should have a production environment of your tooling that you use for your deployments to all environments (from development environment through to production environments), and you will need a development environment of your tooling so that you can continue to test, experiment, and evolve your tooling. You don’t want to do this in an environment that you will use for your next production deployment.

Delivery Model B: Cloud-Enabled Delivery

The cloud-based delivery model leverages a couple of practices that became popular after the continuous-delivery concept was already established. The cloud capabilities became more mature; and environment configuration management tools like Chef, Puppet, and Ansible changed the way we think about creating and managing environments. Together, they also account for the main difference from the previous model: we treat environments and infrastructure like code and hence can build additional environments quickly and reliably. Infrastructure as code means that all infrastructure is defined through configuration, which can be stored in a file and, in turn, can be treated as you would treat the source code of a program.

Figure 9.2: Model B—Cloud-enabled delivery: Cloud-enabled delivery creates a new environment with each deployment

In this model, we create a completely new production environment from scratch, including the applications in their latest version. We can then use this environment to test it with a subset of the production traffic to see whether the changes were successful. If we are happy with the result, we can increasingly move more production traffic to the new environment until no traffic goes to the original environment. At this point, we can destroy the old production environment. This is a really low-risk delivery model, as you can manage the risk by the level of testing of the new environment and by the speed of cutover.

Description of Capabilities

Many of the capabilities are very similar but are just being used in the different context of working with brand-new environments each time. This, for example, eliminates the incremental deployment concern mentioned before, as there is nothing to deploy incrementally to. This means you have to find other ways to deal with persistence between environments (e.g., how do you transition all the transactional data or keep it in a different permanent part of the environment?). This gives an indication of the limitation and complexities with this delivery model and why you might not use it for your whole portfolio of applications.

1. Creating the application: This does not change much between the delivery models.
2. Creating the environment: This is the new capability, and it really means having infrastructure as code for everything other than the application code, which we will deploy later. The required infrastructure includes the compute environments, the network, the operating system, and the middleware. Because we will need details about the infrastructure for the deployment process, you want to make sure you gather the configuration information required. This is very similar to what you often do in a more manual fashion in the continuous-delivery model for your persistent environments. Here, the environments change all the time, so you need to have this part automated. Using environment configuration management with tools such as Puppet or Chef becomes a necessity here, while they were more optional before.
3. Deploying the application: In this model, we have two alternate approaches: we can actively deploy triggered by the creation of a new environment, or we can use the environment configuration-management tools to pull in the right application version. I have seen both models and think that, depending on your preference and context, either one can be chosen. Over time, you will likely end up in the pulling model because it allows you to get rid of a potentially expensive deployment tool and reduce overall complexity in your setup.
4. Testing the application: This will be pretty much the same as in the continuous-delivery model, but you will probably run more tests related to infrastructure because it gets newly created. And you might run a larger regression suite, as you don’t have to take production offline while you do it.
5. Visualizing the delivery process: You have a few more aspects to visualize and measure, such as the number of environments currently in use and the speed and reliability of the environment creation with the new environment creation capability, but the overall ideas remain the same.

Transition Concerns and Organizational Impact

Because the infrastructure is not a separate concern from the overall platform anymore, for this delivery model you should merge your infrastructure team with your platform team. It has become more important for that team to understand automation techniques than to be knowledgeable with Windows or UNIX. You still need those skills, but rather than logging into machines, this team focuses on infrastructure as code.

Mastering the capabilities of the continuous-delivery model is really a prerequisite for this model, as any manual steps in this process diminish the benefits you can get out of this. Additionally, the cloud-based model becomes a lot more beneficial if you change the application architecture to leverage the cloud for elasticity and flexibility. I will discuss this further in chapter 12.

Delivery Model C: Container-Enabled Delivery

The fast rise in popularity of Docker (which has made working with Linux containers a lot easier and brought working with containers into the mainstream) has created this new delivery model that many organizations want to leverage. It works extremely well with a microservice architecture due to the low footprint and flexibility of containers. The speed of this delivery model is impressive, as a new container can be created and deployed in seconds. While the previous delivery models required several minutes to several hours, this is, by far, the fastest model. However, this is only true if you have an architecture with relatively small containers. (If you try to run Siebel or SAP in a container, I suspect the experience will be different.) The immutable nature of containers (at least they should be) will force a lot of good behavior in the organization, as it is not possible to patch the containers manually once they have been created.

Figure 9.3: Container-enabled delivery manages an application in containers

Description of Capabilities

As with the previous model, the capabilities continue to build on top of each other; and all the capabilities built for the previous model can be reused and are, to some degree, prerequisites. The new capabilities have to do with creating and deploying containers.

1. Creating the application: This does not change much between the delivery models.
2. Creating the application container: In addition to the application package that is being stored in the package manager, we are now building application containers that contain everything that is required to run the self-contained application. Some aspects of the automated environment provisioning move into this capability, for example, setting up the required data storage, which is within the container rather than in the environment when deploying microservices. Some people do use an environment configuration-management tool for this purpose, but given the immutability of the container, you can use more lightweight approaches for this one-time build. Container management and governance become a new, crucial capability.
3. Creating the host VM/OS: This is very similar to the “creating the environment” capability. You are building a very simple environment that contains the container engine to which the images will be deployed.
4. Deploying the container: This capability deploys the container into a host and switches it on (e.g., moving traffic to this instance and registering it with the load balancer). This used to be something you had to do yourself, but now there are several tooling solutions to help you with it.
5. Testing the application: This will be pretty much the same as before. Due to the nature of containers, it is very likely that you have more small components in this model, which means more permutations of configuration you could test. Adapting your quality approach for a world of ever-moving configurations will be important for this model to be successful. Remember that all testing is risk management. You will have to come up with a strategy that you are comfortable with, as working with releases in the traditional sense (all changes bundled together over a period of time) is not practical in this model.
6. Visualizing the delivery process: You have a few more aspects to visualize and measure, such as the larger number of containers and their health, in addition to the application health with the new container creation and deployment capabilities; but the overall ideas remain the same.

Transition Concerns and Organizational Impact

Because you are now dealing with immutable containers, the governance of the containers becomes a new organizational responsibility. If new vulnerabilities become known, how do you check where you have used certain libraries so that you can update all container images? Because you build out containers in layers, you could leverage an old image somewhere in the chain or from a public registry that contains known vulnerabilities. You will have to manage templates that you maintain for the organization to manage the number of flavors. Of course, with containers, you can use lots of different technologies at the same time; but as an organization, you want to manage the number of allowed patterns, as you need to maintain your architecture, and individual teams’ preferences for technologies might cause you problems later. Finding the right balance will be something you keep adjusting as you learn more about the usage in your organization.

Similar to the cloud concerns, working with containers means you want to re-architect existing applications to leverage this new paradigm. Just putting your existing application into a large container will not allow you to fully reap the benefits of container-enabled delivery. With this re-architecture activity should also come a reorganizational activity, as it is very inefficient to have an application container owned by more than one team. The best organizational model has the application container fully owned by one team. If the applications are really small, then one team can own multiple applications. If the application container is too large for one team, then it is probably too large in general and should be broken down further. Make Conway’s law work for you this time by creating an organizational structure that you would like to have reflected in your architecture.

Evolving Delivery Model: Serverless Delivery

For those of you who have not heard about serverless architectures, this is a service model where you don’t run servers as such but rather write a function to perform some business logic for you. When the function is called, an instance is created just for the duration of the function call. Amazon Lambda is an example of this architecture. While some of the organizations I work with have experimented with this architecture model, I have not seen wide adoption yet. You might want to investigate the usage of this and find a use-case model to experiment with in your organization.

Capability Roadmap

While there is always a contextual difference between capability roadmaps, I see common patterns to uplift your technical capabilities. You will need to deal with software configuration management (SCM) and application build first in order to reduce the noise, then you will need to follow with application deployments. If you do this in the opposite order, you will see a lot of rework in your deployment automation, as the build artifacts will change once you automate them.

Ideally, you should complete software configuration management and application build and deployment automation together. Test automation requires a somewhat predictable environment to work with (e.g., not one with configuration problems or difference between deployments), so it will benefit from having application build and deployment automated beforehand. Environment provisioning automation tends to have a long lead time, so you can start this in parallel so that it is ready when you need it. And the other capabilities are sitting on top of these foundational capabilities. All of this needs to be supported by the incremental build-out of your DevOps platform to support the automation activities and operational activities, such as monitoring.

In Figure 9.4, I have outlined a common pattern of initial capability uplift. Note that some infrastructure setup and organizational design are required before you start jumping into the technical build-out. The build-out follows the software configuration management/build automation first, the deployment automation next, and then followed by the test-automation pattern, which, in my experience, has the highest chance of success.

Figure 9.4: Sample plan for initial build of capabilities: Container-enabled changes and infrastructure setup are common first steps

First Steps for Your Organization

Map Your Application Delivery Models

As I described above, it is not advisable to push all applications into a container-enabled delivery model, as it would not be economical or feasible. In organizations with a large amount of legacy, you will probably have the largest proportion, targeting continuous delivery and cloud-enabled delivery with some container-enabled delivery in your digital applications. And that is realistic. Remember that the goal is to get better; too often, we make perfect the enemy of better. With this in mind, run a workshop where you review your applications and define what your current and ideal delivery model is for each application. You will need to bring people from your infrastructure, your architecture, and your delivery organization into the same room for this. Then do a fit/gap analysis of the capabilities required for the delivery model you assign to each application. Brainstorm a set of initiatives to build the capabilities that are missing. Often, you can reuse capabilities for applications of the same technology stack (e.g., Java) once they are built for another application. Identify those opportunities for reuse. With all these in mind, define a six-month roadmap, and review the roadmap and progress on a monthly basis to reprioritize based on the lessons learned so far.