Doing something with those metrics

Recording rules and alerts


Prometheus is using the Go Templating System for alerting, in both Prometheus and Alertmanager.

Prometheus splits the alerting role in 3 components:


Alerts and recording rules are close to each other. They are queries that are run at regular interval by prometheus. They both write new metrics into tsdb.


Create, in Prometheus, an alert when a target is down.


Create, in Prometheus, an alert when a grafana server is down, with an extra label: priority=high.


Create a recording rule to get the % of disk space used and alert on > 50% of disk space used.

What is the difference between recording and alerting?

What is an annotation?

What is a “group” of recording rules?

How to see the rules and the alerts in the UI?

What is a pending alert?

Bonus: Alerts unit test (if there is enough time)


Prometheus generates an ALERTS metric with the active/pending alerts.


  1. Download the alertmanager 0.23.0.
  2. Extract it

    $ tar xvf Downloads/alertmanager-0.23.0.linux-amd64.tar.gz
  3. List the files

    $ ls alertmanager-0.23.0.linux-amd64
  4. Launch the alertmanager

    $ cd alertmanager-0.23.0.linux-amd64
    $ ./alertmanager
  5. Open your browser at

  6. Add your alertmanager and your neighbors to prometheus

  7. Connect Prometheus and Alertmanager together

  8. Look for the alerts coming.


Use to get a webhook URL.

Send alerts to that URL.

For the priority=high alerts, send an email instead of a webhook.


How can you check that two alertmanager config are in sync?


There is a alertmanager_config_hash metric


Click to reveal.


Make a big cluster of alert managers


Amtool is the CLI tool for alertmanager

You can use it to e.g. create silences.

$ ./amtool silence --alertmanager.url= add job=grafana priority=high -d 15m -c "we redeploy grafana" -a Julien

That will return the UID of the silence that you can use to expire it.


karma is a dashboard for alertmanager