Deployment overview

The goal of the next two chapters is to establish the supporting infrastructure and resources to deploy Docker applications using AWS. In the spirit of the best practice of defining your infrastructure as code, you will be defining a CloudFormation template that will include all AWS resources required to support your Docker applications running in ECS. As you progress through each chapter, you will build on this template, slowly but surely adding more and more resources until you have a complete solution for deploying your Docker applications in AWS using ECS.

With this in mind, the focus of this chapter is to learn how to build ECS clusters using CloudFormation, and as you have already learned in previous chapters, an ECS cluster is a collection of ECS container instances that you can target when you run an ECS service or ECS task.

ECS clusters themselves are very simple constructs - they simply define a collection of ECS container instances and a cluster name. How these clusters are formed, however, is much more involved and requires several supporting resources, including the following:

• EC2 Auto Scaling group: Defines a collection of EC2 instances with identical configurations.
• EC2 Auto Scaling Launch Configuration: Defines the launch configuration of newly created instances in the Auto Scaling group. A launch configuration often includes user data scripts, which are executed by instances on first run and can be used to trigger the CloudFormation helper scripts you installed in your custom machine image in the previous chapter to interact with CloudFormation Init Metadata.
• CloudFormation Init Metadata: Defines initialization logic that each EC2 instance in the Auto Scaling group should run on initial creation, such as running provisioning commands, enabling services, and creating users and groups. CloudFormation Init Metadata is more powerful that the provisioning capabilities offered by user data, and most importantly, provides a mechanism for each instance to signal to CloudFormation that the instance has successfully provisioned itself.
• CloudFormation Creation Policy: Defines criteria that determines when CloudFormation can consider an EC2 Auto Scaling group as having been created successfully and to continue provisioning other dependencies in the CloudFormation stack. This is based upon CloudFormation receiving a configurable number of success messages from each EC2 instance that is part of the EC2 Auto Scaling group.

There are other ways you can form ECS clusters, but for large-scale production environments, you generally want to use EC2 Auto Scaling groups and use CloudFormation and associated CloudFormation Init Metadata and Creation Policies to deploy your clusters in a robust, repeatable, infrastructure-as-code manner.

How each of these components work together is perhaps best described with a diagram, followed by a short description of how ECS clusters are formed from these components, after which you will proceed to learn how to perform each of the related configuration tasks required to create your own ECS clusters.

The following diagram illustrates the deployment process for creating an ECS cluster, assuming you are using EC2 Auto Scaling groups and CloudFormation:

ECS cluster deployment overview using EC2 Auto Scaling Groups and CloudFormation

In the preceding diagram, the general approach is as follows:

1. As part of your CloudFormation deployment, CloudFormation determines it is ready to start creating the configured ECS cluster resource. The ECS cluster resource will be referenced in CloudFormation Init Metadata that is part of the EC2 Auto Scaling Launch Configuration resource, hence this ECS cluster resource must be created first. Note that at this point, the ECS cluster is empty and is awaiting ECS container instances to join the cluster.
2. CloudFormation creates an EC2 Auto Scaling Launch Configuration resource, which defines the launch configuration that each EC2 instance in the EC2 Auto Scaling group will apply on instance creation. The launch configuration includes a user data script that invokes CloudFormation helper scripts installed on the EC2 instance, which in turn downloads CloudFormation Init Metadata that defines a series of commands and other initialization actions each instance should perform on creation.

3. Once the launch configuration resource has been created, CloudFormation creates the EC2 Auto Scaling group resource. The creation of the Auto Scaling group will trigger the EC2 Auto Scaling service to create a configurable desired number of EC2 instances in the group.
4. As each EC2 instance launches, it applies the launch configuration, executes the user data script, and downloads and executes the provisioning tasks defined in CloudFormation Init Metadata. This will include various initialization tasks, and in our specific use case, the instance will execute the first run script you added to your custom machine image in the previous chapter, to join the configured ECS cluster, ensure the CloudWatch logs agent is configured to log to the correct CloudWatch log groups, start and enable Docker and the ECS agent, and finally, verify the EC2 instance successfully joined the ECS cluster and signal to CloudFormation that the EC2 instance was launched successfully.
5. The Auto Scaling group is configured with a Creation Policy, which is a special feature of CloudFormation that causes CloudFormation to wait until a configurable number of success signals is received from EC2 instances in the Auto Scaling group. Typically, you will configure this to be all instances in your EC2 autoscaling group, ensuring all instances have successfully joined the ECS cluster and are healthy before continuing other provisioning tasks.
6. With the ECS cluster in place with the correct number of ECS container instances derived from the EC2 Auto Scaling group, CloudFormation can safely provision other ECS resources that require a healthy ECS cluster to be in place. For example, you might create an ECS service, which will deploy your container applications to your ECS cluster.