Chapter 9. Troubleshooting Guide

We have come so far and I am sure you are enjoying each and every moment of this challenging and joyful learning journey. I will not say that this book ends after this chapter, but rather you are completing the first milestone. This milestone opens the doors for learning and implementing a new paradigm in the cloud with microservice-based design. I would like to reaffirm that integration testing is an important way to test interaction among microservices and APIs. While working on your sample app Online Table Reservation System (OTRS), I am sure you faced many challenges, especially while debugging the app. Here, we will cover a few of the practices and tools that will help you to troubleshoot the deployed application, Docker containers, and host machines.

This chapter covers the following three topics:

Logging and ELK stack

Can you imagine debugging any issue without seeing a log on the production system? Simply, no, as it would be difficult to go back in time. Therefore, we need logging. Logs also give us warning signals about the system if they are designed and coded that way. Logging and log analysis is an important step for troubleshooting any issue, and also for throughput, capacity, and monitoring the health of the system. Therefore, having a very good logging platform and strategy will enable effective debugging. Logging is one of the most important key components of software development in the initial days.

Microservices are generally deployed using image containers like Docker that provide the log with commands that help you to read logs of services deployed inside the containers. Docker and Docker Compose provide commands to stream the log output of running services within the container and in all containers respectively. Please refer to the following logs command of Docker and Docker Compose:

Note

Docker logs command:

Usage: docker logs [OPTIONS] <CONTAINER NAME>

Fetch the logs of a container:
  -f, --follow    Follow log output
  --help      Print usage
  --since=""                  Show logs since timestamp
  -t, --timestamps            Show timestamps
  --tail="all"                Number of lines to show from the end of the logs

Docker Compose logs Command:

Usage: docker-compose logs [options] [SERVICE...]

Options:

--no-color  Produce monochrome output
-f, --follow  Follow log output
-t, --timestamps  Show timestamps
--tail    Number of lines to show from the end of the logs for each  container
[SERVICES…]  Service representing the container - you can give multiple

These commands help you to explore the logs of microservices and other processes running inside the containers. As you can see, using the above commands would be a challenging task when you have a higher number of services. For example, if you have 10s or 100s of microservices, it would be very difficult to track each microservice log. Similarly, you can imagine, even without containers, how difficult it would be to monitor logs individually. Therefore, you can assume the difficulty of exploring and correlating the logs of 10s to 100s of containers. It is time-consuming and adds very little value.

Therefore, a log aggregator and visualizing tools like the ELK stack come to our rescue. It will be used for centralizing logging. We'll explore this in the next section.

The Elasticsearch, Logstash, Kibana (ELK) stack is a chain of tools that performs log aggregation, analysis, visualization, and monitoring. The ELK stack provides a complete logging platform that allows you to analyze, visualize, and monitor all your logs, including all types of product logs and system logs. If you already know about the ELK stack, please skip to the next section. Here, we'll provide a brief introduction to each tool in the ELK Stack.

Elasticsearch is one of the most popular enterprise full text search engines. It is open sourced software. It is distributable and supports multitenancy. A single Elasticsearch server stores multiple indexes (each index represents a database), and a single query can search data of multiple indexes. It is a distributed search engine and supports clustering.

It is readily scalable and can provide near real-time searches with a latency of 1 second. It is developed in Java using Apache Lucene. Apache Lucene is also free, open sourced, and it provides the core of Elasticsearch, aka the informational retrieval software library.

Elasticsearch APIs are extensive in nature and very elaborative. Elasticsearch provides a JSON-based schema, less storage, and represents data models in JSON. Elasticsearch APIs use JSON documents for HTTP requests and responses.

Logstash is an open source data collection engine with real-time pipeline capabilities. In simple words, it collects, parses, processes, and stores the data. Since Logstash has data pipeline capabilities, helping you to process any event data, like logs, from a variety of systems. Logstash runs as an agent that collects the data, parses it, filters it, and sends the output to a designated app, such as Elasticsearch, or simple standard output on a console.

It is also has a very good plugin ecosystem (image sourced from www.elastic.co):

Logstash

Logstash ecosystem

Kibana is an open source analytics and visualization web application. It is designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices.

It is a browser-based web application that lets you perform advanced data analysis and visualize your data in a variety of charts, tables, and maps. Moreover, it is a zero-configuration application. Therefore, it neither needs any coding nor additional infrastructure after installation.

Generally, these tools are installed individually and then configured to communicate with each other. The installation of these components is pretty straight forward. Download the installable artifact from the designated location and follow the installation steps as shown in the next section.

The installation steps provided below are part of a basic setup, which is required for setting up the ELK stack you want to run. Since this installation was done on my localhost machine, I have used the host localhost. It can be changed easily with any respective host name that you want.

We can install Elasticsearch by following these steps:

  1. Download the latest Elasticsearch distribution from https://www.elastic.co/downloads/elasticsearch.
  2. Unzip it to the desired location in your system.
  3. Make sure the latest Java version is installed and the JAVA_HOME environment variable is set.
  4. Go to Elasticsearch home and run bin/elasticsearch on Unix-based systems and bin/elasticsearch.bat on Windows.
  5. Open any browser and hit http://localhost:9200/. On successful installation it should provide you a JSON object similar to that shown as follows:
    {
  "name" : "Leech",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.3.1",
    "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
    "build_timestamp" : "2016-04-04T12:25:05Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

    By default, the GUI is not installed. You can install one by executing the following command from the bin directory; make sure the system is connected to the Internet:

    plugin -install mobz/elasticsearch-head
  6. Now you can access the GUI interface with the URL http://localhost:9200/_plugin/head/.

    You can replace localhost and 9200 with your respective hostname and port number.

We can install Logstash by following the given steps:

  1. Download the latest Logstash distribution from https://www.elastic.co/downloads/logstash.
  2. Unzip it to the desired location in your system.

    Prepare a configuration file, as shown below. It instructs Logstash to read input from given files and passes it to Elasticsearch (see the following config file; Elasticsearch is represented by localhost and 9200 port). It is the simplest configuration file. To add filters and learn more about Logstash, you can explore the Logstash reference documentation available at https://www.elastic.co/guide/en/logstash/current/index.html.

    Note

    As you can see, the OTRS service log and edge-server log are added as input. Similarly, you can also add log files of other microservices.

    input {
  ### OTRS ###
  file {
    path => "\logs\otrs-service.log"
    type => "otrs-api"
    codec => "json"
    start_position => "beginning"
  }

  ### edge ###
  file {
    path => "/logs/edge-server.log"
    type => "edge-server"
    codec => "json"
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => "localhost:9200"
  }
}
  3. Go to Logstash home and run bin/logstash agent -f logstash.conf on Unix-based systems and bin/logstash.bat agent -f logstash.conf on Windows. Here, Logstash is executed using the agent command. Logstash agent collects data from the sources provided in the input field in the configuration file and sends the output to Elasticsearch. Here, we have not used the filters, because otherwise it may process the input data before providing it to Elasticsearch.

We can install the Kibana web application by following the given steps:

  1. Download the latest Kibana distribution from https://www.elastic.co/downloads/kibana.
  2. Unzip it to the desired location in your system.
  3. Open the configuration file config/kibana.yml from the Kibana home directory and point the elasticsearch.url to the previously configured Elasticsearch instance:
    elasticsearch.url: "http://localhost:9200"
  4. Go to Kibana home and run bin/kibana agent -f logstash.conf on Unix-based systems and bin/kibana.bat agent -f logstash.conf on Windows.
  5. Now you can access the Kibana app from your browser using the URL http://localhost:5601/.

    To learn more about Kibana, explore the Kibana reference documentation at https://www.elastic.co/guide/en/kibana/current/getting-started.html.

As we followed the above steps, you may have noticed that it requires some amount of effort. If you want to avoid a manual setup, you can Dockerize it. If you don't want to put effort into creating the Docker container of the ELK stack, you can choose one from Docker Hub. On Docker Hub there are many ready-made ELK stack Docker images. You can try different ELK containers and choose the one that suits you the most. willdurand/elk is the most downloaded container and is easy to start, working well with Docker Compose.