Matrix testing is a very useful tool if your code needs to be tested in different environments. Depending on your project's needs, the direct support of such a feature in your CI solution may be more or less required.
The easiest way to explain the idea of matrix testing is to take the example of some open source Python packages. Django, for instance, is the project that has a strictly specified set of supported Python language versions. The 1.9.3 version lists Python 2.7, Python 3.4, and Python 3.5 versions as required in order to run Django code. This means that every time Django core developers make a change to the project, the full test suite must be executed on these three Python versions in order to back this claim. If even a single test fails on one environment, the whole build must be marked as failed because the backwards compatibility constraint was possibly broken. For such a simple case, you do not need any support from CI. There is a great Tox tool (refer to https://tox.readthedocs.org/) that, among other features, allows you to easily run test suites in different Python versions in isolated virtual environments. This utility can also be easily used in local development.
But this was only the simplest example. It is not uncommon that the application must be tested in multiple environments where completely different parameters must be tested. The following are a few of these:
- Different operating systems
- Different databases
- Different versions of backing services
- Different types of filesystems
The full set of combinations forms a multi-dimensional environment parameter matrix, and this is why such a setup is called matrix testing. When you need such a deep testing workflow, it is very possible that you require some integrated support for matrix testing in your CI solution. With a large number of possible combinations, you will also require a highly parallelizable building process, because every run over the matrix will require a large amount of work from your building server. In some cases, you will be forced to decide on some trade-off if your test matrix has too many dimensions.
In the next section, continuous delivery is explained.