Alerts: rules and Alertmanager

Why alerts

A dashboard from the previous lesson is useful when someone is looking at it. But at three in the morning nobody is looking, and the service can still be down. An alert is an automatic notification: the system notices a problem in the metrics by itself and wakes the on-call engineer. Without alerts, observability is passive; with alerts, it reacts.

Alert rule

An alert is a condition on a metric that holds for some time. In Prometheus, rules are written in PromQL from the metrics lesson, in a separate YAML file:

groups:
  - name: shop
    rules:
      - alert: HighErrorRate
        expr: >
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "5xx share above 5% for five minutes"

Rules are collected into groups (groups), and each rule has its own fields:

• alert: the alert's name;
• expr: the condition. The alert is active for every series the expression returns;
• for: how long the condition has to hold before the alert fires;
• labels: labels added to the alert. Alertmanager uses them to decide where to send it (severity: critical to a pager, warning to a chat);
• annotations: text for a human: what happened and what to do.

for: 5m cuts off short spikes. Without it, the alert would fire on a one-second blip and wake someone for nothing. When the condition holds for 5 minutes, the problem is real.

The expression above computes the error share for the whole store. If there are several services, sum by (service) (...) is better: then the alert is separate for each service, with a service label, and the on-call engineer immediately sees which one is failing. You can insert this label into the annotation text:

        annotations:
          summary: "{{ $labels.service }}: 5xx share {{ $value | humanizePercentage }}"

$labels holds the series labels, $value holds the expression's value at the moment of the check.

Alert lifecycle

Prometheus checks the rules on the period set by evaluation_interval in its own config (once a minute by default). Every alert has three states:

State	What it means
inactive	the rule is loaded, the condition isn't met
pending	the condition is met, but for hasn't elapsed yet
firing	the condition has held longer than for, the alert goes to Alertmanager

If the condition goes away while the alert is pending, it goes back to inactive and never fires. That's how for damps spikes. The states show up on Prometheus's /alerts page, the loaded rules on /rules.

How Prometheus finds rules

Prometheus learns which files to read from the rule_files section of its config:

rule_files:
  - rules/*.yml

These are path patterns, and a relative path is resolved from the folder where prometheus.yml lives. A pattern that matches no files isn't treated as an error: Prometheus starts and runs without rules. So after an edit, check that the rules were actually loaded. You can see that on /rules or in promtool's output:

promtool check config prometheus.yml

Checking prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: prometheus.yml is valid prometheus config file syntax

Checking rules/alerts.yml
  SUCCESS: 1 rules found

If the rule files found line is missing, no rule file was found at all.

Prometheus rereads changed rules without a restart: on a SIGHUP signal, or on a POST /-/reload request if it's started with the --web.enable-lifecycle flag. If the new file has an error, it keeps the old rules and writes the error to its log.

Alertmanager: what to do with an alert

Prometheus decides that an alert fired, and Alertmanager decides what to do with it: who to send it to, how, when. It's a separate program. Prometheus learns where to send alerts from the alerting section of its config:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

Prometheus (rule fired) → Alertmanager → Slack / PagerDuty / email

Alertmanager handles four things.

• Routing. Different alerts go to different people based on their labels: critical ones to the on-call pager, minor ones to the team chat.
• Grouping. Ten Pods go down at once, and you get one summary notification instead of ten.
• Inhibition. If the "the whole cluster is down" alert is firing, there's no point sending a separate "service not responding" alert for every service in it. An inhibition rule silences the second set while the first one is firing.
• Silence. During planned maintenance, alerts are muted by hand, by a set of labels and for a fixed time, so nobody gets woken up for nothing.

Routes and grouping are set in Alertmanager's config as a route tree:

route:
  receiver: team-chat
  group_by: [alertname, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="critical"]
      receiver: oncall-pager

receivers:
  - name: team-chat
  - name: oncall-pager

An alert travels down the tree and lands in the first route that matches it, or goes to the root receiver (team-chat) if none matched. Real receivers also have channel settings (slack_configs, pagerduty_configs, email_configs, and others), left out here. group_by says which alerts to merge into one notification. group_wait is how long to wait for the rest of the group's alerts before the first notification, group_interval is how often to report new alerts in an already-sent group, repeat_interval is how long to wait before reminding if the alert is still firing. The values in the example match the defaults.

Fighting noise

The main enemy of alerting is noise. If there are too many alerts and some of them are false, the on-call engineer stops reacting to them (alert fatigue), and the important alert drowns. So alerts are kept few and meaningful:

• alert on symptoms, not on every cause: "users are getting errors" matters more than "one Pod has high CPU";
• set a sensible for so you don't catch spikes;
• every alert has to be actionable: if it fires, something needs to be done. If nothing is ever done about an alert, remove it.

Example symptom: "payments are slow." The average response time is misleading here. If 90% of requests are fast and 10% take 2 seconds, the average looks fine, while every tenth user is waiting. That's why a latency alert is set on a percentile (p95, p99), computed from a histogram with the histogram_quantile function from the metrics lesson.

A good set of alerts is often built around the same golden signals and SLOs from the next lesson.

Common mistakes

• Rule not loaded. The file is in the folder, but doesn't match the rule_files pattern, or Prometheus hasn't reread the config. The alert is missing even in the inactive state. Check /rules.
• Condition with no data. A typo in a metric name or label, and the expression returns an empty result. The rule is loaded, there are no errors, the alert stays inactive forever. Run expr without a threshold on the Graph tab.
• Aggregation ate the label you needed. sum(...) without by drops all labels, and the alert comes in without service. It's unclear what to fix.
• No for. The alert flickers on every spike, and the on-call engineer gets used to ignoring it.
• Forgot to remove a silence. A silence was set for a week of planned work, but the work took an hour. Set the duration to match the actual work.

What you'll need for the tasks

• Finding rules on the /rules page and alert states on /alerts in Prometheus.
• The rule_files section in prometheus.yml, and the path patterns in it.
• Reloading the config without a restart (in the labs, bash setup/reload.sh: it runs promtool check config and POST /-/reload).
• Writing a rule: alert, expr, for, labels, annotations.
• histogram_quantile(0.95, sum by (..., le) (rate(..._bucket[1m]))) from the metrics lesson.