Now, let's take a closer look at the tree-based mechanism used by the Alertmanager for routing incoming alerts to a particular receiver. The top-level section of the Alertmanager's configuration file must always define a route block. The block represents the root node of the tree and can contain the following set of fields:
- match: Specifies a set of label values that must match the values from the incoming alert to consider the current route node as matched.
- match_re: Similar to match, with the exception that label values are matched against a regular expression.
- receiver: The name of the receiver to deliver the incoming alert to if the alert matches the current route.
- group_by: A list of label names to group incoming alerts by.
- routes: A set of child route blocks. If an alert does not match any of the child routes, it will be handled based on the configuration parameters of the current route.
To understand how tree-based routing works in practice, let's step through a simple example. For the purpose of this example, the Alertmanager configuration file contains the following routing configurations:
route: receiver: 'default' # All alerts that do not match the following child routes # will remain at the root node and be dispatched to 'default-receiver'. routes: - receiver: 'page-SRE-on-call' match_re: service: cockroachdb|cassandra - receiver: 'notify-ops-channel-on-slack' group_by: [environment] match: team: backend receivers: # omitted: receiver definitions
Let's see how the Alertmanager figures out the appropriate receiver for various incoming alerts by inspecting their label annotations:
- If an incoming alert includes a service label whose value matches either cockroachdb or cassandra, the Alertmanager will dispatch the alert to the page-SRE-on-call receiver.
- On the other hand, if the alert includes a team label whose value is equal to backend, the Alertmanager will dispatch it to the notify-ops-channel-on-slack receiver.
- Any other incoming alert that doesn't match any of the two child routes will be dispatched to the default receiver by default.
This completes our tour of the Alertmanager tool. Granted, configuring alert rules for your applications can, at first, seem like a daunting task. Hopefully, the knowledge you've obtained by reading this chapter will allow you to begin experimenting with Prometheus and set up a few rudimentary Alertmanager test rules. With a little bit of practice and once you get the hang of the rule syntax, you will find that writing more sophisticated rules for monitoring your production applications will become a breeze!