Metrics and Prometheus

Why this matters

Metrics are numbers over time: how many requests per second, how much memory is in use, how many responses ended in an error. Someone has to collect them, store them, and feed them to graphs and alerts. The standard for this in the cloud world is Prometheus. It's a time series database plus a collection system built specifically for metrics.

Without metrics, "is the service unhealthy?" gets answered by user complaints. With metrics you can see that errors started climbing at 14:05, and memory hit its limit a minute before that.

The pull model

Prometheus's main feature is that it goes and fetches metrics itself. Services don't send anything; they just keep a page with their current values, and Prometheus periodically picks it up. This is the pull model. It keeps services simple (they don't need to know about Prometheus) and gives Prometheus control over how often it collects.

A service that wants to expose metrics runs an HTTP endpoint (by convention /metrics) and serves its current values there in a simple text format:

$ curl http://app:8080/metrics
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1530
http_requests_total{method="GET",status="500"} 12
process_resident_memory_bytes 4.12e+08

The # TYPE line says what type the metric is, # HELP describes it in words. After that comes one line per label combination: name, labels in curly braces, value.

One such trip for metrics is called a scrape. The scrape target is the address Prometheus collects metrics from. Targets are listed in the prometheus.yml config or discovered automatically through service discovery (for example, every Pod in Kubernetes with the right annotation):

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: api
    static_configs:
      - targets: ["api:8080"]

job_name groups targets that serve the same purpose. The default path is /metrics, a different one is set with the metrics_path field. The scrape interval defaults to one minute if not set explicitly; typical configs use 15-30 seconds.

The port in targets has to be the one the service actually serves metrics on. Many services serve them on a separate port, not the one that handles regular requests.

Target state and the up metric

For every target, Prometheus writes a metric called up on its own: 1 if the last scrape succeeded, 0 if it didn't. It adds the labels job (from job_name) and instance (the target's address) to each series.

The list of targets with their state shows up on the Status → Targets page (http://<prometheus>:9090/targets). The Error column there gives the reason a target is DOWN: connection refused, a timeout, or an HTTP 404 if there's no metrics page at that address.

Metric types

Prometheus has four types:

• Counter — only goes up (or resets to 0 on restart). Request counts, error counts, processed jobs. By convention the name ends in _total. The raw value is almost useless; you compute a rate from it instead: "requests per second".
• Gauge — can go up and down. Current memory, active connection count, queue length. A snapshot of "right now", read as is.
• Histogram — a distribution across buckets. Request latency: how many landed within 100 ms, how many within 500 ms. One histogram produces several series: _bucket with a le label (how many values are at most the bucket boundary), _sum (the sum of all values), and _count (how many values). Percentiles (p95, p99) are computed by Prometheus itself with the histogram_quantile function, from buckets already collected across all instances.
• Summary — also about distributions, but the service itself computes the percentiles and hands over finished numbers. You don't need to pick buckets, but you also can't compute anything new on top: ready-made percentiles can't be combined across instances. You can't get a cluster-wide p99 out of a summary.

Histogram and summary solve the same problem from different angles. If the metric comes from one instance and picking buckets is too much hassle, a summary is enough. When you need aggregation across the whole service or flexibility in queries, use a histogram. In practice, cloud services almost always use histograms.

Here's what a histogram looks like on the /metrics page:

http_request_duration_seconds_bucket{le="0.1"} 120
http_request_duration_seconds_bucket{le="0.5"} 180
http_request_duration_seconds_bucket{le="+Inf"} 200
http_request_duration_seconds_sum 31.4
http_request_duration_seconds_count 200

120 requests landed within 0.1 s, 180 within 0.5 s, 200 requests total. The buckets are cumulative: each one includes all the smaller ones.

Labels

A metric can have labels: key-value pairs that slice one metric into dimensions:

http_requests_total{method="GET", status="200", path="/api"}

Labels let you ask "how many 500 errors were there on /api" without creating a separate metric for every case. But labels have a trap. Every unique combination of label values is a separate time series. A status label with five values and a path label with twenty give up to a hundred series for a single metric.

You can't put something with a huge number of possible values into a label (a user ID, an email, a URL with parameters): the series count grows into the tens and hundreds of thousands, Prometheus eats memory and slows down. This is called high cardinality. The Status → TSDB Status page helps find the culprit: it lists the metrics and labels with the most series and values.

Queries: PromQL

You pull data out with the PromQL language. The simplest query is a metric name with a label filter. It returns the current value of every matching series:

process_resident_memory_bytes{job="api"}

That's enough for a gauge. A counter needs a rate over a window:

rate(http_requests_total{status="500"}[5m])

This means "500 errors per second, averaged over the last 5 minutes". [5m] is the window; it needs at least two scrapes inside it, otherwise rate returns nothing. Labels in curly braces filter series: = is an exact match, =~ is a regular expression (status=~"5.." matches all 5xx).

If there are several series, you add them up with sum, and by (...) keeps the labels you need:

sum by (status) (rate(http_requests_total[5m]))

A percentile from a histogram is computed over its buckets; you must keep the le label when summing:

histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

These queries are the basis for graphs (usually in Grafana, that's the next lesson) and alerts: "if the error rate stays above a threshold for 5 minutes straight, wake the on-call engineer".

Common mistakes

• Reading a counter like a gauge. http_requests_total equal to 35,000 tells you nothing by itself: the counter has been accumulating since startup. Use rate. And the other way around, rate over a gauge gives a meaningless number.
• A DOWN target and empty graphs. The wrong port or path is set in targets. First open Status → Targets and read the Error column.
• IDs in labels. user_id, email, a full URL with parameters in a label blow up the series count. That kind of data belongs in logs, not in metrics.
• A rate window shorter than two scrape intervals. With a 15-second scrape interval, rate(...[15s]) returns nothing. Use a window at least a few times longer than the interval.
• Adding up summary percentiles. The average or sum of p99 from three instances is not the service's p99. You need a histogram and histogram_quantile.
• A short-lived job without Pushgateway. A script that finishes in a second may never get scraped by Prometheus. Pushgateway exists for that case, but long-running services don't need it.

What you'll need for the tasks

• Open the Prometheus web interface: the Graph page, Status → Targets, and Status → TSDB Status.
• Read the prometheus.yml config: job_name, targets, metrics_path, and restart Prometheus after an edit (docker compose restart).
• Check what a service serves on /metrics, and tell a counter apart from a gauge and a histogram by the # TYPE line.
• Write PromQL queries: filter by labels, rate, sum by, histogram_quantile, count.