CHAPTER 8
Implementing J&S Food's DevOps Platform Using AWS PaaS
DevOps and its resulting technical, architectural, and cultural practices represent a convergence of many philosophical and management movements (including): Lean, Theory of Constraints, Toyota production system, resilience engineering, learning organizations, safety culture, Human factors, high-trust management cultures, servant leadership, organizational change management, and Agile methods.
—Gene Kim
The stake in this second phase of J&S Food's operating model digitalization is to equip the company with a digital products and services development platform that guarantees online customers a superior digital food experience.
The challenge, as you know, is not to implement the technology; it is rather to set up an ecosystem of people, processes, practices, and values supported by technology in a way that enables the company to deliver innovative products and services continuously.
The implementation of such an ecosystem is not easy. It involves mobilizing and teaming up a variety of people, tools, methodologies, and skills. This takes time and energy.
The safest solution is to rely on a framework that can simplify and speed up DevOps implementation.
In this chapter, in the role of the chief information officer (CIO), you'll be involved in a workshop to define the architecture and implementation of the platform that will underpin J&S Food's digital product and service development platform.
In this meeting, led by the AWS Solutions Architect and the enterprise architect, you'll be part of the dialogue involving the VP of marketing, sales representatives, customer service, developers, testers, and IT operations. The goal is to ensure that the platform to be implemented meets the stakeholders' vision.
Transformation Journey's Third Stage: Implementing J&S Food's DevOps Platform
The second phase of J&S Food's digital business model digitization is about implementing DevOps. As discussed in Chapter 4, “Industrializing Digital Product and Service Development,” DevOps is the combination of software development (Dev) and IT operations (Ops) practices and tools designed to increase the organization's ability to deliver applications and services faster.
The bottom line is to get a digital product and service development platform that enables the expected digital food experience.
Figure 8.1 illustrates where the digital transformation team stands in the business model digitization.
Figure 8.1: The DevOps implementation phase
As Figure 8.1 shows, the AWS Solutions Architect would address the DevOps implementation effort in these three steps:
- Understanding what J&S Food is trying to achieve
- Understanding DevOps in the AWS computing environment
- Implementing J&S Food's digital product and service development platform
Let's see how the Solutions Architect would go about implementing J&S Food's digital product and service development platform.
Understanding What J&S Food Is Trying to Achieve
In the next day's workshop, the Solutions Architect starts by saying this:
The VP of marketing replies as follows:
Jotting down the technological concerns (AI and IoT), the Solutions Architect asks for further clarification:
The VP of marketing delivers the requested clarification as follows:
The consultant asks the following:
As the project leader on the business side, the VP of marketing replies:
The next sections explain why DevOps is the platform that will help meet J&S Food's digital business expectations and more importantly how to implement it.
Understanding the DevOps Implementation in the AWS Computing Environment
As you must realize, what the VP of marketing expects is much more than a technical and technological platform that will miraculously deliver the digital products and services expected by customers.
The challenge for the Solutions Architect and the enterprise architect is to set up a consistent ecosystem of people, processes, practices, and tools that allows the delivery of J&S Food's digital food experience.
This section discusses the whys and wherefores of the DevOps implementation in the AWS computing environment including the common challenges and mistakes and, more importantly, the DevOps implementation framework for AWS.
Discussing the Challenges
Seeking to engage the implementation team, the Solutions Architect says the following, provocatively:
Trying to clear up the development team's apparent confusion, the enterprise architect clicks the mouse to display the diagram shown in Figure 8.2.
Figure 8.2: DevOps as part of the digital business value chain
The enterprise architect then clarifies:
The Solutions Architect adds to the enterprise architect's explanations as follows:
Understanding the Common Mistakes
Proceeding with the dos and don'ts, the Solutions Architect makes the following clear:
Anticipating the participants' questions about that broad approach of DevOps implementation in the AWS computing environment, the Solutions Architect says the following:
The DevOps Implementation Framework for AWS
Clicking the mouse to move to the presentation's next page, the DevOps blueprint for AWS is displayed, as illustrated in Figure 8.3.
The Solutions Architect explains the following:
Figure 8.3: The DevOps implementation framework for AWS
The Solutions Architect then adds this:
The Solutions Architect next elaborates on the three primary building blocks of the DevOps implementation framework for AWS.
The Digital Business Value Chain Primary Activities and AWS Modern Application Influence
The Solutions Architect reminds the participants of the following:
The purpose of the primary activities in the framework is to help, through a questionnaire, software development stakeholders to agree on the software development activities necessary for the implementation of digital products. Two important points to know are the role of the AWS modern application characteristics and that of the change management function.
Be aware that AWS modern application development (MAD) refers to the performance enablers discussed in Chapter 4. Their role is to recommend software engineering best practices for the AWS computing environment to ensure software delivery in line with the quality requirements of the company within ever shorter timeframes.
As for the software engineering best practices for AWS, its role is to infuse and consolidate in the delivery pipeline, technical and operational best practices that make the delivery process continuously efficient.
Table 8.1 illustrates the digital business primary activity specification matrix.
Table 8.1: The Digital Business Primary Activity Specification Matrix
| SOFTWARE DEVELOPMENT ACTIVITY #1? | SOFTWARE DEVELOPMENT ACTIVITY #2? | SOFTWARE DEVELOPMENT ACTIVITY #3? | |
|---|---|---|---|
| Purpose | What's the purpose of the activity? | Same question as activity #1 | Same question as activity #1 |
| Stakeholders and roles | Who are the stakeholders? What's the role of each? | ||
| MAD: Culture of ownership | What actions must be taken? | ||
| MAD: Architectural patterns | How well to use microservices? | ||
| MAD: Computing in modern applications | How well to use containers and Lambda functions? | ||
| MAD: Data management | How can data management leverage microservices to address scaling, fault tolerance, and speed? | ||
| MAD: Developer agility | What are the best practices to deploy to increase productivity? | ||
| MAD: Operational model | How to leverage serverless to make IT and application operations agile? | ||
| MAD: Management and governance | How to leverage programmatic guardrails that make governance checkpoints that slow down innovation? |
The AWS CodePipeline Service
The Solutions Architect states the following:
CodePipeline is an AWS continuous delivery service that enables you to model, visualize, and automate the steps required to release your software. The CodePipeline role in the DevOps implementation framework is to act as the DevOps pipeline.
The DevOps pipeline is the set of automated processes and tools that allows developers and operations to collaborate on building and deploying code to the production environment.
Two main components make up AWS CodePipeline: the delivery stages and the toolchain. Let's discuss them now.
Understanding AWS CodePipeline Delivery Stages
The AWS CodePipeline delivery stages refer to the software delivery lifecycle (SDLC) discussed in Chapter 4.
Stages including Source, Build, Stage, and Production are built-in features of the AWS CodePipeline service. Each stage contains actions that are performed on the application artifacts. Artifacts can be anything ranging from built, tested applications or updated containers.
Table 8.2 sums up the purpose of the AWS CodePipeline stages.
Table 8.2: Functions of the AWS CodePipeline Stages
| PURPOSE | DESCRIPTION | |
|---|---|---|
| Source | Manages changes to software | In this stage, three functions are performed: creating the development environment, tracking the changes in software, and committing changes to the changes repository. |
| Build | Manages the continuous integration and continuous deployment (CICD) process | In this stage, four functions are automated: detecting new release in the changes repository, building the software (compiling and linking), testing, and deploying to the stage environment. |
| Stage | Manages acceptance tests | In this stage, four functions are performed: user acceptance test (UAT), operational acceptance test (OAT), acceptance tests approval, and deployment to production. |
| Production | Operates production environment and applications | In this stage, three primary functions are automated: production environment and application monitoring, operations logs analysis, and operational practices continuous improvement. |
Note that the steps outlined in Table 8.2 are not set in stone. Other steps may be introduced depending on the needs of the business.
Defining AWS CodePipeline Toolchain
The AWS CodePipeline toolchain refers to the combination of tools mobilized across the delivery pipeline to implement stage functions.
Table 8.3 summarizes the type of tools supporting the functions of the delivery pipeline stages.
Table 8.3: The Technology Mobilized Across the Delivery Pipeline Stages
| TYPE OF SOLUTIONS | EXAMPLES OF MARKET SOLUTIONS | |
|---|---|---|
| Source | Version control system (VCS) and integrated development environment (IDE) | Git and AWS CodeCommit (VCS), Eclipse (IDE) |
| Build | Continuous integration (CI), build, and unit test | Jenkins (CI), Ansible and Maven (Build), and Junit (Unit Test) |
| Stage | User acceptance test (UAT) and operational acceptance test (OAT) automation | Selenium and IBM Rational Functional Test (automated UAT) and JMeter (automated OAT) |
| Production | Cloud monitoring and management solution | AWS CloudWatch, Nagios, and Splunk |
Links to DevOps tools in the AWS environment are provided in the reference section of this chapter.
Understanding DevOps Implementation for AWS
The implementation of the AWS DevOps blueprint depicted in Figure 8.4 is a three-step process.
Figure 8.4: The DevOps implementation process for the AWS computing environment
These steps include specifying the software development activities, mapping the development activities to the stages of the delivery pipeline, configuring and creating the AWS CodePipeline platform, and deploying the primary activities of the digital business value chain. Let's discuss them.
Specifying the Software Development Activities
The Solutions Architect states the following:
This activity consists of identifying the software development tasks that will constitute the value chain. The objective is to specify these software development tasks in terms of the actors to involve as well as the necessary best practices recommended by the AWS modern application likely to make them efficient.
The digital business activity specification matrix is a useful tool that allows software development stakeholders to capture, evaluate, and agree on the software development tasks to select.
Mapping the Development Activities to the Delivery Pipeline Stages
This activity is about mapping the software development tasks to the actions that are likely to automate them in the AWS CodePipeline platform. The bottom line is to associate each software development task with the corresponding stage in the delivery pipeline and then identify the AWS services or AWS partner services likely to automate them.
An example of an action and its related AWS service is the EC2 instance creation and deployment and the related AWS service Elastic Beanstalk, which is invoked to provision development, testing, or production environments. Another example of an action is the installation of Maven, an open source build tool for enterprise Java projects designed to take much of the hard work out of the build process.
A last example of an action is the build software that invokes the external Maven service to build (compile and link) the software in the build stage of the delivery pipeline.
Configuring and Creating the AWS CodePipeline
This task consists of configuring the delivery pipeline from the AWS CodePipeline console wizards or from the CloudFormation designer. The challenge is to automate the software delivery process from end to end. For each stage of the pipeline, including source, build, staging, and production, the following essential information is configured:
- The desired action
- The AWS service or AWS partner service to invoke
- The AWS service or partner service configuration information
Following the configuration of the production stage, the AWS CodePipeline delivery pipeline is ready to process software releases.
Implementing J&S Food's Digital Product and Service Development Platform
The objective is to design and implement a platform that addresses the technological, organizational, and operational issues.
This section discusses the design of J&S Food's DevOps toolchain and agile operating model.
J&S Food's Digital Product and Service Development Platform
Figure 8.5 illustrates the architecture of J&S Food's digital product and service development platform.
Figure 8.5: J&S Food's digital product and service development platform
The platform is based on the AWS CodePipeline service and is structured around two components: digital business primary activities and associated best practices and the toolchain.
The Solutions Architect discusses how the platform works:
J&S Food’s DevOps Toolchain
J&S Food's toolchain mobilizes four environment types including source, build, staging, and production.
The following sections outline the configuration of these stages.
The Source Environment
The source environment is the set of compute infrastructure and tools necessary for the software coding and unit testing activities. It's the environment that the development team will leverage to develop the ecommerce website, the smart shopping bag software components, and the two-sided marketplace application.
Table 8.4 sums up the automated action and related services of the source environment.
Table 8.4: The Source Platform's Actions and AWS Services
| ACTIONS | PURPOSE | AWS/PARTNER SERVICES |
|---|---|---|
| Create development environment | Creates the extended EC2 instances intended to support the development environments for the ecommerce website, two-sided marketplace, and smart shopping bag applications | AWS Elastic Beanstalk |
| Install AWS toolkit | Installs in the extended EC2 instances the complete AWS toolkit for Microsoft Visual Studio | AWS Toolkit for Visual Studio |
| Commit changes | Posts changes in ecommerce website, two-sided marketplace, and smart shopping bag software to the Git repository | Git integrated with AWS CodePipeline via a plugin |
The Build Environment
The build environment is the platform that will be generated by the AWS CodePipeline service to support the automation of the continuous integration and continuous delivery process.
Table 8.5 summarizes the actions that will be orchestrated to achieve the CICD process for the ecommerce website, two-sided marketplace, and smart shopping bag applications.
The Staging Environment
The staging environment is the platform that will be generated by the AWS CodePipeline service to automate the user acceptance test and operational acceptance test processes for each of the ecommerce website, two-sided marketplace, and smart shopping bag applications.
Table 8.5: The Build Platform's Actions and AWS Services
| ACTIONS | PURPOSE | AWS/PARTNER SERVICES |
|---|---|---|
| Create the build environment | Triggered once changes in software are committed to the Git repository. This creates the environment intended to support the CICD processes. | AWS Elastic Beanstalk |
| Build the applications | Compiles, links, and generates executable files for the ecommerce website, two-sided marketplace, and smart shopping bag applications. | Maven integrated with AWS CodePipeline via a plugin |
| Perform unit tests | Performs automated unit tests for the ecommerce website, two-sided marketplace, and smart shopping bag applications. | Selenium integrated with AWS CodePipeline via a plugin |
| Deploy applications | Deploys the ecommerce website, two-sided marketplace, and smart shopping bag applications to the staging environment. | AWS Elastic Beanstalk |
Table 8.6 outlines the actions that will be orchestrated to achieve the acceptance test for the ecommerce website, two-sided marketplace, and smart shopping bag applications.
Table 8.6: The Staging Platform's Actions and AWS Services
| ACTIONS | PURPOSE | AWS/PARTNER SERVICES |
|---|---|---|
| Creates staging environment | Creates the extended EC2 environments needed to perform user acceptance tests and operational acceptance tests | AWS Elastic Beanstalk |
| Performs user acceptance tests (UATs) | Performs automated functional tests | Selenium Functional integrated with AWS CodePipeline via a plugin |
| Performs operational acceptance tests (OATs) | Performs automated performance tests | Selenium Non-Functional integrated with AWS CodePipeline via a plugin |
| Approval | Approves acceptance tests report and deployment to production of the ecommerce website, two-sided marketplace, and smart shopping bag applications | Non applicable |
The Production Environment
The production environment refers to the setting where the ecommerce website, two-sided marketplace, and smart shopping bag applications will be made live and operated for use by the intended users.
Table 8.7 sums up the actions and AWS and partners services that will be leveraged to deliver the expected J&S Food digital experience.
Table 8.7: The Production Platform's Actions and AWS Services
| ACTIONS | PURPOSE | AWS/PARTNER SERVICES |
|---|---|---|
| Creates production environment | Creates the extended EC2 environments that will host and operate the ecommerce website, two-sided marketplace, and smart shopping bag applications | AWS Elastic Beanstalk |
| Measures digital food experience | Analyzes the extended EC2 logs to evaluate J&S Food's digital food experience from the IT operations perspective | Splunk Functional integrated with AWS CodePipeline via a plugin |
| Continuously improves operations practices | Finds operations dysfunction, identifies the root causes, corrects the root causes | Same as above |
J&S Food’s Digital Business Primary Activities
Discussions about J&S Food's digital business primary activities concluded that five primary activities would support the development of three applications. Table 8.8 sums up the primary activities mobilized for each application along with the recommended AWS modern application best practices to focus on.
Table 8.8: The Primary Activities of J&S Food's Business Model
| ACTIVITY | BIG DATA APPLICATIONS | ECOMMERCE WEBSITE | TWO-SIDED MARKETPLACE |
|---|---|---|---|
| Business requirements analysis | Discuss requirements with the product owner. UML use case or agile user story is the recommended best practice. | Same as big data applications | Same as big data applications |
| Software architecture design | Discuss the software architecture. Modular architecture based on microservices combining serverless and container is recommended. | Same as big data applications | Same as big data applications |
| Software coding | Java based on Microsoft Visual Studio Code is the recommended language. | Same as big data applications | Same as big data applications |
| Software testing | Automated CI based on Test-Driven Development (TDD) using a Selenium solution is the recommended practice. | Same as big data applications | Same as big data applications |
| Application deployment | Continuous deployment to production based on automatic UAT and OAT approvals is the recommended practice. | Same as big data applications | Same as big data applications |
Designing the Agile Operating Model
The goal is to make the chain of primary activities agile by intelligently and appropriately deploying the principles and philosophy of agile methodologies such as Scrum and Extreme Programming (XP).
However, the Solutions Architect might temper the development team's enthusiasm as follows:
The Usual Challenges and Solutions
In the following sections, the enterprise architect discusses the do's and don'ts of agile operating model deployment.
Understanding the Challenges
The enterprise architect reminds the participants of the following:
In addition to these old cultures and mentalities, the challenge will be to use a deployment approach that simplifies issues like defining accountabilities, interactions across the delivery pipeline stages, and rapid decision-making structures.
The enterprise architect provides the most effective approach:
Understanding the Solutions
Agile methodologies are helpful as they provide actionable organizational and operational frameworks. These frameworks offer software development lifecycles that underpin delivery pipelines. They define roles and responsibilities, interactions between stakeholders, and also decision-making structures across the software delivery lifecycle.
The next section describes the agile operating model that resulted from the decision to base J&S Food's DevOps platform on Scrum and XP.
J&S Food’s Agile Operating Model Defined
Figure 8.6 illustrates the agile operational model piece of J&S Food's DevOps platform.
Figure 8.6: J&S Food's next agile operating model
Figure 8.6 shows that the Scrum approach facilitates software development and product launch activities. Scrum clarifies roles and responsibilities; through daily Scrums, this approach supports collaboration and decision-making.
Table 8.9 summarizes the roles and responsibilities in J&S Food's next digital product and service development team.
Table 8.9: J&S Food's Digital Product and Service Development Team
| ROLES AND RESPONSIBILITIES | |
|---|---|
| Product owner | Defines stories (expected software functions), prioritizes product backlog (set of business requirements), and anticipates acceptance criteria for acceptance tests |
| Scrum master | Facilitates interactions, the Scrum events, including daily Scrums and sprint review, to accelerate the development effort and guarantee quality |
| Developer | Leverages the DevOps toolchain and AWS modern application principles to deliver high-quality software |
| Tester | Oversees and monitors automated tests during the build and staging phases of the software development effort |
| IT operations | Relies on AWS CloudWatch, Nagios, and Splunk services to monitor the DevOps toolchain and guarantee superior customer experience |
The enterprise architect praises the benefits of Scrum and XP methodologies to the participants:
Daily Scrums are meetings held daily involving the software development stakeholders. The purpose of such meetings is to set the context for the coming day's work. Benefits of daily Scrums include the following:
- Increased collaboration and ownership
- Improved progress visibility and exposure
- Increased control of the software development effort
- Reduced risk
- High quality
Sprint reviews are the meetings that take place at the end of the sprint. A sprint is a time-boxed period where the Scrum team works to complete a set amount of work. The purpose of the sprint review is to evaluate the latest features and to consider the plan for the software in the future.
As to the sprint retrospective, it's an “improvement” meeting held to identify improvements in all aspects of the software development process. The meeting is attended by the product owner, Scrum master, development team members, and other relevant stakeholders.
Key Takeaways
This chapter demonstrated that DevOps is a catalyst of the digital business performance provided that its implementation considers operational and organizational matters.
In this chapter, you learned about the big picture of DevOps, which integrates, as a whole, technological, organizational, and, more importantly, business performance issues through the AWS Modern Application.
In the next chapter, we will complete the study of the DevOps implementation by addressing the innovation services that J&S Foods will need to develop its lead digital product: the smart shopping bag.
References
- 1. Ernest Mueller, “What is DevOps and How Does it Work?” The Agile Admin (August 02, 2010).
https://theagileadmin.com/what-is-devops/ - 2. Amazon Web Services, “AWS CodePipeline: Automate Continuous Delivery Pipelines for Fast and Reliable Updates,” Amazon Web Services (November 22, 2017).
https://aws.amazon.com/codepipeline/ - 3. Aditya Sridhar, “An Introduction to Git: What is it, and How to Use it?” FreeCodeCamp (August 12, 2018).
https://www.freecodecamp.org/news/what-is-git-and-how-to-use-it-c341b049ae61/