Rolling out a broken release: rollback and postmortem

A classic evening release: CI reported success, kubectl rollout status came back with successfully rolled out, and a couple of minutes later 5xx errors started pouring in. The first question of an incident isn't "why is the code broken" but "how fast can we bring back the last version". The cause gets investigated after the rollback.

Deployment is a Kubernetes object that keeps a set number of identical Pods running. Pod is the smallest unit you run, a container lives inside it. ReplicaSet tracks the number of Pods for one version of the template. kubectl is the command-line tool for managing a cluster.

Revision history

Every change to spec.template creates a new ReplicaSet and bumps the revision number. A revision is a saved version of the Pod template at the moment of the change. Editing fields outside the template, replicas for example, doesn't create a revision. By default a Deployment keeps 10 old ReplicaSets, the number comes from spec.revisionHistoryLimit.

kubectl rollout history deploy/web

deployment.apps/web
REVISION  CHANGE-CAUSE
1         release v1
2         release v2
3         release v2

The CHANGE-CAUSE column comes from the kubernetes.io/change-cause annotation. An annotation is a free-form key-value note on an object, set by hand or from CI:

kubectl annotate deploy/web kubernetes.io/change-cause="release v3" --overwrite

The annotation sits on the Deployment and gets copied into every new revision. In the example above, revision 3 was rolled out without updating the note, so it inherited it from revision 2. If no annotation was ever set, the column shows <none>, and during an incident you're left guessing what each revision carried. Either way, the note is just a note: it says nothing about whether the revision actually works.

The template of a specific revision:

kubectl rollout history deploy/web --revision=2

deployment.apps/web with revision #2
Pod Template:
  Labels:	app=web
	pod-template-hash=...
  Annotations:	kubernetes.io/change-cause: release v2
  Containers:
   web:
    Image:	busybox:1.36.1
    ...
    Environment:
      APP_VERSION:	v2

The command shows the image, run command, env, volumes and probes of that revision. Handy for comparing versions before a rollback.

Rollback

To the previous revision:

kubectl rollout undo deploy/web

To a specific one:

kubectl rollout undo deploy/web --to-revision=1

A rollback doesn't create a new ReplicaSet. It copies revision 1's template into the Deployment, and the Deployment reuses that old ReplicaSet, bumping its number. After rolling back from revision 3 to revision 1, the history looks like this:

REVISION  CHANGE-CAUSE
2         release v2
3         release v2
4         release v1

Revision 1 is gone, and revision 4 shows up with the same template instead. The history only grows, numbers never wind back.

Here's the trap. undo with no arguments goes to the previous revision, and that one might be broken too. Running undo again brings you right back: the command just bounces between the last two revisions. If two bad releases went out in a row, you need --to-revision and a template check with --revision=N.

If the Deployment was created with kubectl apply, the rollback warns you:

Warning: resource deployments/web was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, ...

A rollback changes the object in the cluster, but not the manifest in Git. The next kubectl apply of the old file rolls the broken version out again. After the incident, put the working version back into the manifest too.

A rollback is a rolling update like any other, just in reverse. The failed ReplicaSet shrinks to zero, the working one grows back to the full replica count. Speed depends on maxSurge and maxUnavailable. If the working ReplicaSet still has ready Pods alive, the rollback finishes almost instantly.

Check the result like this:

kubectl rollout status deploy/web --timeout=120s
kubectl get pods -l app=web
kubectl get deploy web -o jsonpath='{.spec.template.spec.containers[0].image}'

Why readiness didn't stop the rollout

readinessProbe is a check that decides whether a Pod is ready to take traffic. A rollout brings up new Pods in small batches, the batch size comes from maxSurge and maxUnavailable (25% of the replica count by default). An old Pod is removed only once a new one becomes Ready. If a new Pod fails the readinessProbe, the rollout gets stuck, and the old Pods stay alive and keep serving traffic.

The service breaks in two cases.

First: there's no readinessProbe at all. Then kubelet marks the Pod ready right after the container starts. kubelet is the Kubernetes agent on a node that runs containers. The process crashes a few seconds after starting, but the rollout already counted the Pod as ready and removed the old one.

Second: the readinessProbe checks the wrong thing. For example, httpGet on / answers 200 as long as the web server is alive, even if the app can't connect to the database. The rollout succeeds, while the real endpoints return 500. This kind of check is exactly what most often lets a broken release through to production.

If a new Pod doesn't start at all (a bad image tag, ImagePullBackOff, or the process crashes with readinessProbe enabled, CrashLoopBackOff), the rollout never finishes. kubectl rollout status waits until all new Pods become ready:

Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...

A correct readinessProbe

The check should hit a health endpoint that confirms the app is actually ready, not just that a web server process is alive.

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

The path here is just an example, every app has its own. What matters is that the handler checks whatever the app needs to serve a request, a database connection, for example.

initialDelaySeconds: 5 gives the app time to start before the first check. Too high a value slows down the rollout: the Pod waits longer for Ready status. Too low is safe for readiness, the first checks simply fail and the Pod becomes Ready a bit later.

periodSeconds: 5 means a check every 5 seconds.

failureThreshold: 3 means three failures in a row clear the ready mark, and the Service drops the Pod from its endpoints. Endpoints are the list of Pod IP addresses the Service sends traffic to.

Check dependencies only in readinessProbe. If livenessProbe checks the database too, a database outage makes kubelet restart every Pod at once, and that doesn't help.

A slow start (Java, Rails, warming up caches) is handled by a separate startupProbe. That's a check that waits for the app's first startup:

startupProbe:
  httpGet:
    path: /readyz
    port: 8080
  failureThreshold: 30  # 30 × periodSeconds = 5 minutes to start
  periodSeconds: 10

startupProbe disables liveness and readiness until it succeeds once. After that, readiness works as usual.

maxSurge and maxUnavailable

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%

maxSurge is how many Pods are allowed above the replica count during a rollout. Percentages round up. The higher the value, the faster the rollout and the higher the load on the cluster.

maxUnavailable is how many of the required replicas are allowed to be not ready during a rollout. Percentages round down. With 4 replicas, 25% gives one Pod: the rollout is allowed to remove it without waiting for a new one. With 3 replicas, the same 25% rounds down to zero.

With maxUnavailable: 0, the cluster never loses capacity: the rollout moves slower, but new Pods that fail the readinessProbe can't take away a single old one. Both fields can't be zero at the same time, or the rollout would have nothing to move with.

Common mistakes

Blind undo. The previous revision might be broken too, especially if a hotfix went out right after a failed release. First kubectl rollout history, then --revision=N, then undo --to-revision=N.

Trusting CHANGE-CAUSE. The note is set by hand, people forget it or copy it. The template tells you what actually works, not the note.

Editing the probe before rolling back. Changing the readinessProbe creates a new revision based on the current, broken template. A rollback after that brings back the old check. Roll back first, then edit.

rollout status without --timeout in CI. The step hangs for up to ten minutes, then fails with an error that doesn't mean much to a human.

How to avoid this next time

• maxUnavailable: 0 for critical services.
• readinessProbe on a health endpoint that checks dependencies, not on /.
• a kubernetes.io/change-cause annotation on every release, so the revision history is readable during an incident.
• kubectl rollout status --timeout=... in every rollout pipeline.

Useful commands

kubectl rollout history deploy/web
kubectl rollout history deploy/web --revision=2
kubectl rollout undo deploy/web
kubectl rollout undo deploy/web --to-revision=1
kubectl rollout status deploy/web --timeout=120s
kubectl get rs -l app=web

kubectl explain deployment.spec.strategy.rollingUpdate
kubectl explain pod.spec.containers.readinessProbe
kubectl explain pod.spec.containers.startupProbe

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-broken-rollout     # set up the starting state
./run.sh verify k8s-broken-rollout    # check the current step

In short

• Roll back first, investigate after: kubectl rollout undo brings back the last ReplicaSet.
• kubectl rollout history --revision=N shows the image, env, volumes and probes of a specific revision. Pick a revision by its template, not its note.
• A successful rollout status doesn't mean a working service if there's no readinessProbe, or it hits /.
• maxUnavailable: 0 keeps a broken release from taking capacity away from Pods that still work.
• A slow start is handled by startupProbe, not by a bloated initialDelaySeconds on liveness.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl rollout history deploy/<name> and --revision=N.
• kubectl rollout undo deploy/<name> --to-revision=N and kubectl rollout status deploy/<name> --timeout=....
• kubectl get deploy <name> -o jsonpath='{...readinessProbe}', kubectl edit deploy <name> or kubectl patch.
• kubectl exec deploy/<name> -- wget -qO- -T 5 http://...: a request from inside a Pod.