Incidents and postmortems

Why this matters

No matter how much you invest in reliability, services still go down sometimes. From the SLO lesson you know that 100% doesn't happen, and the error budget is a direct admission of that. You need to both prevent outages and know how to recover from them: react fast, then learn the lesson. That's what incident management and postmortems are for.

Concepts

Incident — an unplanned disruption of a service: users see errors, data doesn't arrive, everything works too slowly.

Incident commander — the person who coordinates the work during an incident: assigns tasks, decides what to do next, makes sure everyone knows the current state. They usually don't fix anything themselves.

Mitigation — an action that stops the damage even if the cause hasn't been found yet: rolling back a release, switching traffic, adding resources.

Postmortem — a written analysis of an incident after it's closed.

Blameless — the principle of a postmortem with no blame. You look for how the system allowed the outage to happen, not who's at fault.

Incident lifecycle

An incident has clear stages:

1. Detection. An alert fired or users reported it. The earlier the better, and that needs alerts from the observability-basics track.
2. Response. Someone takes charge of the incident (the incident commander), pulls in the right people, coordinates.
3. Mitigation. Bring the service back first, investigate later. Roll back the release, switch traffic, add resources. The goal of this stage is to stop the damage; the root cause gets found later.
4. Resolution. The service is healthy again.
5. Postmortem. A calm analysis afterward.

An important principle: during an incident you fix, you don't investigate. Roll back first, analyze later.

Here's why. While you're reading the new release's code and hunting for the bug, users keep getting errors. Rolling back to a version that worked yesterday takes minutes and is almost always safe. Fixing forward can take an hour and introduce a new bug. That's why the new release isn't deleted, it's kept around for the postmortem: that's where you'll look for the cause.

A rollback in Docker Compose looks like this: in compose.yml you change the image tag to the previous one and recreate the service.

services:
  web:
    image: registry.example.com/shop:1.8.2

docker compose up -d

docker compose up -d compares the configuration with the running container and recreates it if something changed. docker compose restart can't do that: it restarts the same container with the old settings, edits to compose.yml don't get applied.

Roles in a major incident

At Google SRE, work during a major incident is split between roles so people don't get in each other's way:

• Incident command: coordinates, keeps the overall picture, hands out the other roles.
• Operational work: the only ones who change the system (roll back, switch traffic).
• Communication: regularly posts status for users and the rest of the company.
• Planning: tracks the long-running tasks, writes down what's been done and everything that will go into the postmortem.

On a small team, one or two people can cover all of this. What matters is that one person coordinates, and everyone knows who that is.

Blameless postmortem

A postmortem is written after the incident is closed. Its key word is blameless (no blame). The point of the analysis is to understand how the system allowed the outage to happen, and what to change so it doesn't happen again. You don't look for someone to blame.

Why no blame? If people get punished for a mistake, they start hiding details and dodging an honest analysis, and the organization stops learning. When there's no blame, an engineer can calmly explain what they did and why, and the real cause comes out. Someone pressed the wrong button not because they're a bad engineer, but because the system let them press the wrong button with no safeguard.

A postmortem usually contains:

• What happened: a couple of sentences a person outside the team can understand.
• Timeline: events with exact times, from the first event that led to the outage through the recovery.
• Impact: who couldn't do what, how many requests or how much money was lost, how long it lasted.
• Root cause: why the outage became possible.
• Actions: what to change, each action with an owner and a deadline.

Building a timeline from logs

You build the timeline from what was actually recorded, not from memory: service logs, alert history, the deploy journal, messages in the on-call chat. Write times in one timezone, usually UTC, otherwise events from different sources won't line up.

The timeline immediately gives you two useful numbers. How long from the first errors to the alert (how fast it was noticed). And how long from the alert to recovery (how fast it was fixed). If the alert came half an hour after the first errors, one of the action items will be about alerting.

Example timeline:

- 14:02:10  deployed shop 1.9.0
- 14:02:31  first 500s in the shop log
- 14:09:00  alert on 5xx share, on-call engineer paged
- 14:12:40  rolled back to shop 1.8.2
- 14:13:05  5xx share back to zero

Here the errors lasted 11 minutes, almost 7 of them before the alert.

Working on causes, not symptoms

A good postmortem gets down to the root cause instead of stopping at the surface. The "five whys" technique: ask "why" several times in a row.

The service crashed. Why?
→ It ran out of memory. Why?
→ A leak in the new release. Why?
→ There was no memory limit or growth alert. Why?
→ The service template didn't include them by default.

The action from an analysis like that isn't "fire the release's author", it's "add a memory limit and alert to the service template". That way one outage fixes a whole class of future ones.

The number five is just a rule of thumb. You stop once you've reached something you can change in the system: a template, a procedure, a check in CI. If the chain ends at a person ("the engineer made a mistake"), ask once more: why did the system let that mistake reach users?

Common mistakes

• Investigating instead of rolling back. Half an hour of reading code while users see 500s. Bring back the working version first.
• Deleting the broken release. The new release's image and logs are needed for the analysis. Roll back without deleting anything.
• Losing the logs. docker compose up -d recreates the container, and docker compose logs only shows the new one. The old container's logs disappear with it. Save what you need before the recreate, or write logs somewhere that survives the container.
• docker compose restart after editing compose.yml. The container restarts with the old settings. New settings need docker compose up -d.
• Building the timeline from memory. A day later, nobody remembers the exact times. Take the times from logs and alerts.
• Action items with no owner or deadline. Nobody will do "improve monitoring". Someone will do "add an alert on the 5xx share, owner so-and-so, by Friday".
• A postmortem with someone to blame. The next incident will be hidden.

What you'll need for the tasks

• docker compose ps, docker compose logs <service>, docker compose up -d (from the lab's setup/ folder).
• docker inspect and the Config.Env field, docker compose exec <service> cat <file>.
• curl -i http://127.0.0.1:<port>/: the response code and headers.
• The "Incident lifecycle" section: rolling back to the previous image tag.
• From the secrets lesson: secrets: in Compose and reading a password from the file /run/secrets/<name>.
• The "Blameless postmortem" and "Working on causes" sections: the postmortem's structure and "five whys".

What to remember

Outages are inevitable, so it matters that you know how to work through them. An incident goes through stages from detection to postmortem. During an incident you first stop the damage (rollback, traffic switch), and investigate afterward.

A postmortem is blameless: the goal is to understand how the system allowed the outage, not to punish a person. Punishment kills an honest analysis. You build the timeline from logs, get down to the root cause, and fix the system, not the symptom.

This closes out the ops-basics track: secrets and incident analysis. The mini-project at the end of the lesson ties both topics together.