Metrics, logs, traces

Why this matters

While a service is small, it's enough to SSH in and look at the log. Once there are dozens of services spread across machines, that stops working. You need to see what's happening inside the system without climbing into every copy by hand. That's what observability is.

Observability rests on three types of signals: metrics, logs, traces. Each answers its own question, and swapping one for another is awkward.

Metrics: what and how much

A metric is a number over time. How many requests per second, what's the latency, how much memory is used, how many errors. Metrics are cheap: the system stores aggregates over time, not every event.

A service usually serves metrics as text over HTTP. Here's what the Prometheus format looks like:

http_requests_total{route="/api/cart",status="200"} 1530
http_requests_total{route="/api/cart",status="500"} 12
http_request_duration_seconds{route="/api/cart",quantile="0.95"} 0.24
process_resident_memory_bytes 4.12e+08

The curly braces hold labels: they slice a single metric into rows, for example by route and response code. quantile="0.95" means p95: 95% of requests finished within this time, here 0.24s.

Metrics answer the question "what's happening and how bad is it". They're used to build graphs and alerts: "if latency is above 500ms for five minutes, wake the on-call engineer." But a metric won't say why it's slow. It's just a number.

Logs: what exactly happened

A log is a record of a specific event with details. A metric says "there were 12 errors", a log shows which ones:

2026-06-18T10:14:02 ERROR order=4821 payment failed: gateway timeout
2026-06-18T10:14:05 ERROR order=4822 payment failed: gateway timeout

Logs answer the question "what exactly happened". A metric showed a spike in errors, the logs showed it was timeouts from the payment gateway. There are a lot of logs, so they're filtered by time, service, and severity level.

Traces: where the chain gets stuck

In a system of many services, one request passes through several: gateway, orders, payments, database. A trace shows the whole path of one request and how much time it spent in each service.

A trace is made of spans. A span is one operation in one service: "gateway accepted the request", "payments service charged the card." All spans of a request share a trace_id, and every span except the first (the root) has a parent. That makes a tree:

trace abc123  total 240ms
  gateway     POST /api/pay     240ms
    orders    create order      225ms
      payments  charge card     195ms   ← this is where it's slow
      db        INSERT order      5ms

A parent's time includes its children's time. orders takes 225ms, but 200 of that is waiting on payments and db; it spends about 25ms on its own. So you look for the culprit not by the longest row, but by the service that spends time on its own.

A trace answers the question "where exactly in the chain is the bottleneck". A metric shows that requests are slow, a trace points to the service for a specific request.

How they work together

The signals complement each other through the steps of an investigation:

1. Metric alert: "latency went up." Now you know there's a problem.
2. Trace: "the payments service is slow." Now you know where.
3. Logs from the payments service: "gateway timeout." Now you know why.

For step 3 to work, services write trace_id into every log entry. Then the identifier from the trace finds the entries for exactly this request, not hundreds of similar ones:

{"level":"error","service":"payments","trace_id":"abc123","msg":"charge failed","reason":"gateway_timeout"}

The identifier is passed between services in a request header (the W3C Trace Context standard, the traceparent header). Today all three signals are usually collected through OpenTelemetry: an open standard and set of libraries for traces, metrics, and logs.

Common mistakes

• Looking only at errors. The alert is about latency, but attention drifts to the route with 5xx codes. A slow request can still finish successfully.
• Blaming the longest span. A parent span is never shorter than its children. Look at who spends time on its own.
• Taking the cause from the log of the first service you find. The gateway will write "slow response", but that's a symptom. The cause is written by the service that's actually slow.
• Logs without trace_id. Without it a log can't be connected to a trace, and the investigation turns into guesswork by timestamp.
• Collecting one signal instead of three. Metrics alone won't say why, logs alone won't show the scale.

What you'll need for the tasks

• Read metrics in Prometheus format: labels in {}, quantile="0.95" as p95.
• Read a span tree and figure out which service spends time on its own.
• docker compose -f <file> exec <service> wget -qO- <URL>: run an HTTP request from inside a container.
• docker compose -f <file> logs --no-log-prefix <service> and grep on a quoted field ('"trace_id":"..."'), a pipeline of several greps.

In the next lesson we'll cover log levels and structured logs, then metrics and Prometheus in more detail.