Metrics Monitoring

Querying Prometheus

Metrics monitoring is different because it does not assume that a situation can be explained in fixed states. Instead, it brings you inside your system and provides dozens of metrics that you can then analyze and understand.

Even if you do not need all the metrics, it is better to collect them to look further what they look like.

What is a metric


Labels are used to add metadata to metrics. That can be used to differenciate then, e.g. by adding a status code, a URI handler, a function name, …

Types of metrics

Metric that can go up and down
Metric that can only go up and starts at 0
Metrics that put data into buckets. An histogram is composed of 3 metrics: sum, buckets and count.
Metrics that calculate quantiles over observed values. A summary is composed of 3 metrics: sum, quantiles and count.

Upstream documentation


The number of http requests is expressed in …

The duration in …

The number of active sessions is a …

What are the labels added by prometheus ?

What are the labels prometheus knows but does not add?

Labels matching

For prometheus to do calculations and comparisons of metrics, labels must match one each side (except __name__, the name of the metric)


Some important functions:

In the list above, which ones should be used with counters, and which ones with gauges?

What is the difference between irate and rate? idelta and delta?


How can I get the sum of scraped metrics by job (2 ways)? – exclusion and inclusion.


The scrape_samples_scraped metric is added by prometheus after sraping a target and indicates the number of metrics for that job at that scrape.


Click to reveal.

How can I get the % of handler="/federate" over the other prometheus_http_request_duration_seconds_count ?


Click to reveal.

Over Time

What is the difference between max and max_over_time?

Max, Min, Bottomk, Topk

What is the difference between max(x) and topk(1, x)?

Time functions



How to use the optional argument of day_of_week?

What is the timestamp() function? How can it be useful?


You can use Grafana Explore feature to get help and autocomplete on Prometheus (currently still a “beta” feature). That feature will likely come in the next release of Prometheus!


Can you think of any usecases for and/or/unless?