SLI, SLO, and error budget

Why this matters

You can't verify that "the service is working well." That's a feeling, not an engineering statement. To talk about reliability in concrete terms, you measure it, and there are three concepts for that: SLI, SLO, and error budget. They turn the vague word "reliable" into numbers you can base decisions on: ship a release today, or work on stability first.

This lesson wraps up the module. The Prometheus metrics, Grafana dashboards, and alerts from earlier lessons come together here into one system: the SLI is computed from metrics, drawn on a dashboard, and an alert fires on budget burn.

SLI: what we measure

SLI (Service Level Indicator) is a metric of service quality from the user's point of view. Not "CPU is at 60% load," but something the user actually feels:

• the share of successful requests (not 5xx);
• the share of requests faster than a threshold, say under 300 ms;
• availability over a period.

It's convenient to write the SLI as the share of "good" events among all events. That way it always falls between 0 and 1, and it's easy to compare against a target.

SLI = good requests / all requests

What counts as a good request is decided in advance. A 5xx response means the service failed. That's a bad event. A 404 for an order that doesn't exist is the service responding correctly; the client made the mistake. That's why 4xx usually doesn't count as a failure. Put it in the "bad" bucket and the SLI will show an outage where there isn't one.

SLI in PromQL

The metrics from the Prometheus lesson already have everything you need. The http_requests_total counter with a status label gives you both the numerator and the denominator:

sum(rate(http_requests_total{job="api", status!~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))

There are three details here, and skipping any one of them breaks the SLI.

• status!~"5.." is a regular expression. The =~ (and !~) operator compares against a regex, while = looks for an exact match. status="5.." looks for a status that is literally equal to the string 5.., and finds nothing. A regex in PromQL is anchored to the start and end of the value, so 5.. matches 503 but not 1503.
• sum(...) on both sides. Without it, Prometheus divides series pairwise by matching labels, and instead of one number you get a series per response code.
• The [5m] window is the same in the numerator and the denominator. Otherwise the share is computed over different time spans.

Expressions like this get long, and nobody rewrites them in every panel. They're saved as a recording rule: Prometheus evaluates the expression itself on every rule cycle and writes the result into a new series with a readable name. The documentation recommends names of the form level:metric:operations, for example sli:availability:ratio_rate5m.

groups:
  - name: api-slo
    rules:
      - record: sli:availability:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="api", status!~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))

SLO: what target to set

SLO (Service Level Objective) is a target for the SLI: what value counts as good enough over a period. Not 100%, but a realistic bar.

SLI: share of successful requests
SLO: 99.9% successful requests over 30 days

99.9% sounds almost perfect, but over 30 days that's about 43 minutes during which requests can fail. SLO is set below 100% on purpose. Absolute reliability costs an infinite amount, and the user won't notice the difference between 99.9% and 100% anyway: their own Wi-Fi and mobile network already fail more often than that.

Don't confuse SLO with SLA. SLA (Service Level Agreement) is a contract with the client that spells out what happens on a breach: a penalty, a discount, a refund. SLO is an internal engineering target. It's usually set stricter than the SLA, so the team notices a problem before any payouts start.

Error budget: license to take risks

If the SLO is 99.9%, the remaining 0.1% is the allowed share of failures. It's called the error budget. It's a license to fail that you can spend.

error budget = 1 - SLO
SLO 99.9% → budget 0.1% → 30 days × 24 h × 60 min × 0.001 ≈ 43 minutes

The same calculation for other targets:

SLO	Budget over 30 days
99%	432 min (7.2 h)
99.9%	43.2 min
99.99%	4.32 min

Each extra nine shrinks the budget by a factor of 10. For a request-based SLI, "minutes" is a convenient fiction: the budget is really in requests. At 99.9%, out of every million requests, 1,000 are allowed to fail.

The budget turns reliability into a decision-making tool:

• budget available (fewer failures than the norm): you can take risks, ship more often, try new things;
• budget exhausted: risky changes get postponed and the team works on stability.

This settles the age-old argument where development wants to ship faster and operations wants stability. While there's budget, you ship. Once it's gone, you stabilize. The decision is made by the number. What exactly to do once the budget runs out, the team writes down in advance in an error budget policy.

Burn rate: how fast the budget is spent

Knowing how much budget is left isn't enough. You also need to know how fast it's draining. That speed is called burn rate:

burn rate = current error share / error share allowed by SLO

A burn rate of 1 means the service is spending its budget at exactly the rate that uses it all up by the end of the 30-day window. A burn rate of 5 means the whole month's budget will be gone in 6 days. Example: SLO 99.9% allows 0.1% errors, and the service is currently returning 0.5%. Burn rate = 0.5 / 0.1 = 5.

In PromQL, it's convenient to compute burn rate on top of the SLI recording rule:

(1 - sli:availability:ratio_rate5m) / (1 - 0.999)

Connecting to alerts

SLO tells you what to alert on. The alert from the previous lesson is most useful when it's tied to budget burn. "The budget is burning 14 times faster than normal" matters more than a raw threshold on a metric. That's an alert on the symptom the user feels, not on the cause.

- alert: ApiErrorBudgetBurn
  expr: (1 - sli:availability:ratio_rate1h) / (1 - 0.999) > 14.4
  for: 2m
  labels:
    severity: page

Where 14.4 comes from. The Google SRE Workbook suggests paging a human if 2% of the monthly budget burns in an hour. 2% of 30 days packed into 1 hour gives a burn rate of 0.02 × 720 / 1 = 14.4. Slow burn (10% of the budget over 3 days, burn rate 1) becomes a ticket, not a 2 a.m. phone call. In production, alerts like this are usually built on two windows at once, a long one and a short one. The long window confirms the budget really is being spent, and the short one lets the alert clear quickly once the fix lands.

In this lesson's labs the windows are shorter (1 minute instead of an hour, for in seconds), so you see the result right away. The formulas are the same.

Common mistakes

• 4xx counted as bad requests. The SLI drops because of client errors, and the team spends its budget on something it can't fix. Bad events are what the service itself is responsible for.
• = instead of =~ in the status filter. status="5.." finds nothing. The numerator is empty, and the whole expression returns an empty result, not zero. An alert on an expression like that stays silent forever, and Prometheus still shows the rule's status as ok: the syntax is valid, there's just no data.
• Dividing without sum. Instead of one SLI you get a series per label, and some of them equal 1.
• SLO of 100%. There's no budget, any failure breaks the target, which means you can't ship or experiment at all.
• An alert without for. It fires on a one-second spike, and the on-call engineer stops trusting alerts.

What you'll need for the tasks

• Prometheus UI: the Graph tab (run an expression), Alerts (alert state), Status → Rules, and Status → Targets.
• Recording rule and alerting rule in a YAML rules file (record:, alert:, expr:, for:, labels:).
• Label filters =, !=, =~, !~, and the rate() and sum() functions.
• Reloading the Prometheus config without a restart: POST /-/reload (in the labs, bash setup/reload.sh does this along with a promtool check).
• Budget calculation: 30 × 24 × 60 × (1 − SLO) minutes, burn rate as the ratio of the error share to 1 − SLO.
• For the mini-project: scrape_configs in prometheus.yml, a Grafana dashboard JSON model, and docker compose logs.