Continuous integration and testing

Continuous integration and testing are other important aspects of the development cycle. Continuous integration clubs the codes together, and testing will make sure that the code which goes into production is bug free, and that most of the issues are mitigated in a lower environment. There are many different way of testing: unit testing, integration testing and system testing. We will be integrating this testing in an automated pipeline in the example. I will be using the examples provided by Google Cloud with a simple hello world function, and running them in an automated way, and then putting them into a pipeline for single-touch deployment. 

I have created a HelloWorld function referencing from Google's existing example repository. I have put them on my local repository, which we will be using for setting continuous integration and testing. I have also created a Dockerfile, which will help us to create a docker container for Jenkins, pre-installed gcloud, Node.js and the functions emulator which we will be using to set up our DevOps automation:

1. Git clone the Git repository mentioned as follows. We are cloning this locally to get the Dockerfile, and you can play around with it by making changes to the script and testing the deployment:

$ git clone https://github.com/shzshi/google-functions-helloworld.git

2. We will build a Docker image locally and then start a Jenkins portal to enable us to set up the automation. You need to make sure Docker is installed on your local machine for this example.
3. We move to the Dockerfile directory:

$ cd google-functions-helloworld

4. We create a Docker image with Jenkins, gcloud, a function emulator, Node.js and all other required libraries:

$ docker image build -t google-functions .

5. Here, run the Docker container with the image  created in the previous line. The Docker container will be hosted locally with port 8080 and 50000 exposed. We will also map the volume with a local host directory:

$ docker run --rm -it -p 50000:50000 -p 8080:8080 -v /My/Local/Host/PATH/chapter6/google-functions/jenkins:/var/jenkins_home google-functions:latest

6. Once the container is running, we will browse the Jenkins portal through http://localhost:8080. If you are creating this container for the first time without mapped volume, you will be asked to copy and paste the password and install a default plugin. 
7. Log in to the Jenkins portal using the credentials you had created earlier, or you already had. Then click on New Item. 
8. Enter the item name as my-serverless-google-functions, select the freestyle project and then click on OK.
   

9. Go to the tab Source Code Management and select Git, then copy and paste the repository mentioned below into the Repository URL textbox and leave the rest as default: https://github.com/shzshi/google-functions-helloworld.git.
10. We need to create a Google service account for gcloud within Jenkins to authenticate with GCP:
    • Go to the OPEN THE LIST OF CREDENTIALS (https://console.cloud.google.com/apis/credentials?_ga=2.77044693.-1734735492.1524930885) page
    • Click on Create Credentials
    • Select the Service Account Key
    • Click on the drop-down for Service account and select New Service Account
    • Enter a name for the service account in Name
    • Use the default Service account ID or generate a different one
    • Select the Key type: JSON
    • Click Create, the Service account created window is displayed, and the private key for the Key type you selected is downloaded automatically to our local machine which we will be using further.
    • Click Close
11. You need to fork your repository and copy the content of the gcloud service account key JSON file into the file My-Serverless-Project-1d8bacd4886d.json, because we will be using this JSON file in Jenkins to authenticate: https://github.com/shzshi/google-functions-helloworld.git.
12. Go to the Build tab, and from the drop-down menu's Add build step select Execute Shell and then add the following steps into the Command text area:

gcloud auth activate-service-account --key-file=${WORKSPACE}/My-Serverless-Project-1d8bacd4886d.json
 gcloud config set project ${YOUR_GCP_PROJECT_ID}
 npm install
 
 export NODE_PATH=${WORKSPACE}/node_modules


# executing unit test
 ${WORKSPACE}/node_modules/.bin/ava test/unit.http.test.js

 # executing integration test
 export BASE_URL=http://localhost:8010/${YOUR_GCP_PROJECT_ID}/${YOUR_GCF_REGION}
${WORKSPACE}/node_modules/.bin/functions start
${WORKSPACE}/node_modules/.bin/functions deploy helloHttp --trigger-http
${WORKSPACE}/node_modules/.bin/ava test/integration.http.test.js
 # deploying to GCP project and executing system test
 gcloud beta functions deploy helloHttp --trigger-http
export BASE_URL=https://${YOUR_GCF_REGION}-${YOUR_GCP_PROJECT_ID}.cloudfunctions.net/helloHttp
${WORKSPACE}/node_modules/.bin/ava test/system.http.test.js

13. We need to parameterize the job, which means that we need to add two text parameters, one for the gcloud project id and another for the gcloud region. Add the parameters as mentioned in the following screenshot. The Default Value needs to be changed with your project name and region:

14. Once everything looks fine, as mentioned in the previous 13 steps, click SAVE, and save the project. 
15. To run the job, click Build with Parameters and run the job with a default parameter. 
16. If the job runs successfully, we should see the following output and therefore Google Function has successfully passed the unit, integration and system tests:

Building in workspace /var/jenkins_home/workspace/my-serverless-google-functions
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/shzshi/google-functions-helloworld.git # timeout=10
Fetching upstream changes from https://github.com/shzshi/google-functions-helloworld.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/shzshi/google-functions-helloworld.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision bbfb5c17a65dab7f0a8e0b3dc3de82e5792fe21d (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bbfb5c17a65dab7f0a8e0b3dc3de82e5792fe21d
Commit message: "added function emulator"
 > git rev-list --no-walk bbfb5c17a65dab7f0a8e0b3dc3de82e5792fe21d # timeout=10
[my-serverless-google-functions] $ /bin/sh -xe /tmp/jenkins2470859504083131546.sh
+ gcloud auth activate-service-account --key-file=/var/jenkins_home/workspace/my-serverless-google-functions/My-Serverless-Project-1d8bacd4886d.json
Activated service account credentials for: [jenkins@my-serverless-project-201920.iam.gserviceaccount.com]
+ gcloud config set project my-serverless-project-201920
Updated property [core/project].
+ npm install
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.3 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

up to date in 30.041s
+ export NODE_PATH=/var/jenkins_home/workspace/my-serverless-google-functions/node_modules
+ /var/jenkins_home/workspace/my-serverless-google-functions/node_modules/.bin/ava test/unit.http.test.js

   helloHttp: should print a name
   helloHttp: should print hello world

  2 tests passed

+ export BASE_URL=http://localhost:8010/my-serverless-project-201920/us-central1
+ /var/jenkins_home/workspace/my-serverless-google-functions/node_modules/.bin/functions start
Warning: You're using Node.js v8.11.1 but Google Cloud Functions only supports v6.11.5.
Starting Google Cloud Functions Emulator...
Google Cloud Functions Emulator STARTED
┌────────┬───────────┬─────────┬──────────────────────────────────────────────────────────────────────────┐
│ Status │ Name      │ Trigger │ Resource                                                                 │
├────────┼───────────┼─────────┼──────────────────────────────────────────────────────────────────────────┤
│ READY  │ helloHttp │ HTTP    │ http://localhost:8010/my-serverless-project-201920/us-central1/helloHttp │
└────────┴───────────┴─────────┴──────────────────────────────────────────────────────────────────────────┘
+ /var/jenkins_home/workspace/my-serverless-google-functions/node_modules/.bin/functions deploy helloHttp --trigger-http
Warning: You're using Node.js v8.11.1 but Google Cloud Functions only supports v6.11.5.
Copying file:///tmp/tmp-1653qKGF3wXgKZdC.zip...
Waiting for operation to finish...done.
Deploying function............done.
Function helloHttp deployed.
┌────────────┬──────────────────────────────────────────────────────────────────────────┐
│ Property   │ Value                                                                    │
├────────────┼──────────────────────────────────────────────────────────────────────────┤
│ Name       │ helloHttp                                                                │
├────────────┼──────────────────────────────────────────────────────────────────────────┤
│ Trigger    │ HTTP                                                                     │
├────────────┼──────────────────────────────────────────────────────────────────────────┤
│ Resource   │ http://localhost:8010/my-serverless-project-201920/us-central1/helloHttp │
├────────────┼──────────────────────────────────────────────────────────────────────────┤
│ Timeout    │ 60 seconds                                                               │
├────────────┼──────────────────────────────────────────────────────────────────────────┤
│ Local path │ /var/jenkins_home/workspace/my-serverless-google-functions               │
├────────────┼──────────────────────────────────────────────────────────────────────────┤
│ Archive    │ file:///tmp/tmp-1653qKGF3wXgKZdC.zip                                     │
└────────────┴──────────────────────────────────────────────────────────────────────────┘
+ /var/jenkins_home/workspace/my-serverless-google-functions/node_modules/.bin/ava test/integration.http.test.js

   helloHttp: should print a name (110ms)
   helloHttp: should print hello world (104ms)

  2 tests passed

+ gcloud beta functions deploy helloHttp --trigger-http
Deploying function (may take a while - up to 2 minutes)...
.............done.
availableMemoryMb: 256
entryPoint: helloHttp
httpsTrigger:
  url: https://us-central1-my-serverless-project-201920.cloudfunctions.net/helloHttp
labels:
  deployment-tool: cli-gcloud
name: projects/my-serverless-project-201920/locations/us-central1/functions/helloHttp
serviceAccountEmail: my-serverless-project-201920@appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/gcf-upload-us-central1-39234144-2d9a-4a19-bf5f-82ef8d19a71b/5433a10a-52ed-4330-b68a-f3c3be016035.zip?GoogleAccessId=37314512027@cloudservices.gserviceaccount.com&Expires=1525049193&Signature=C3H2oyjrieTuykum%2BH09BqZO63bCXE4vlrWVzOhEMTBAoPAUFTHU96JVqTk7ZiGdl2iv34a7FlR70vE9oo4jnnZApVtCYHVSY9JA3X%2BAn4VR4Aw510UUC7ilZbGGJ5U3eyk1bVlzQxTkx20Mq6yx8JUQqRMj%2FTisHqs0MCHC9k83NJj6JQdF%2BbgVLzPg%2Bfrm06kZzhKqKmLQ7XOMXwSHlm%2F74N6%2B5JyByPvtPlHqgGoIdW8X7eyys4%2B22X3zU0Z3MjXG7emlA9t4Hrpa3oQ0AUiSD78b1Mnfbz%2FYXdj%2BKDY4fjv5JQcOBZLj8DEw8sbcSdJcIvvAKrPwcKy7Y1eiMg%3D%3D
status: ACTIVE
timeout: 60s
updateTime: '2018-04-30T00:16:35Z'
versionId: '2'
+ export BASE_URL=https://us-central1-my-serverless-project-201920.cloudfunctions.net/helloHttp
+ /var/jenkins_home/workspace/my-serverless-google-functions/node_modules/.bin/ava test/system.http.test.js

   helloHttp: should print hello world (371ms)
   helloHttp: should print a name (1.3s)

  2 tests passed

Finished: SUCCESS

So, through the previous example, we were able to build, test and deploy the function locally and on gcloud. But we did this just randomly on a project. Imagine that we need to have multiple environments, and performance systems, and performance testing with very minimal manual intervention. We need to set up a pipeline with approval gates which deploys to multiple environments, mitigating most of the problems before the function is actually in production. This is where continuous delivery comes in handy.