Functional tests part deux – testing in production!

That's not to say that you cannot actually run your functional tests in a live production environment! Surely whether that's a good or bad idea is a debatable point, but if you do decide to go down that route, there are a few patterns that you can apply to achieve this in a safe and controlled way.

To get the ball rolling, you can begin by revising your DB schemas so that they include a field that indicates whether each row contains real data or is part of a test run. Each service could then silently ignore any test records when it handles live traffic.

If you are working with a microservice architecture, you can engineer your services so that they do not talk to other services directly but rather to do so via a local proxy that is deployed in tandem with each service as a sidecar process. This pattern is known as the ambassador pattern and opens up the possibility of implementing a wide range of really cool tricks, as we will see later in this chapter.

Since all the proxies are initially configured to talk to the already deployed services, nothing prevents us from deploying a newer version of a particular service and have it run side-by-side with the existing version. Since no traffic can reach the newly deployed service, it is common to use the term dark launch to refer to this kind of deployment.

Once the new versions of the services that we need to test against have been successfully deployed, each functional test can reconfigure the local proxies to divert test traffic (identified perhaps by an HTTP header or an other type of tag) to the newly deployed services. This can be seen in the following diagram:

Figure 4: Using the ambassador pattern to test in production

This neat trick allows us to run our tests in production without interfering with live traffic. As you can tell, live testing requires substantially more preparation effort compared to testing in a sandbox. This is probably one of the reasons why QA teams seem to prefer using staging environments instead.

In my view, if your system is built in such a way that you can easily introduce one of these patterns to facilitate live testing, you should definitely go for it. After all, there is only so much data that you can collect when running in an isolated environment whose load and traffic profiles don't really align with the ones of your production systems.