As load balancing can introduce extra complexities into your environment, it's important to ensure that you can test your setup thoroughly. Especially when you're trying to debug corner cases, being able to build a test platform becomes critical to reproducing faults or issues.
Ideally, you want this combined with real-time metrics as well, especially if you want to be able to overlay this with data from your upstream servers. As we'll cover in Chapter 13, NGINX Plus – The Commercial Offering, the commercial release of NGINX contains a live monitoring module which provides information such as the current connections, upstream server statuses, and load information.
While there are a lot of programs and cloud services around which you can generate load to test your load balancer from a client perspective, I didn't find many tools to have test instances for the upstream side of the problem. So, like any programmer, I simply wrote my own!
HTest is an open source tool, which emulates a web server under varying conditions. It is written in Go and is therefore able to take advantage of the high levels of concurrency and optimization; HTest can serve more than 150,000 requests a second on very modest hardware.
Rather than just serving a static page, HTest allows you to vary the response times (so that you can emulate your typical application responses), failure rates (where a percentage of responses return a 500 error), and can also introduce some jitter so that the results are more realistic.