Deployment rollback

Where Kubernetes keeps the release history, and what kubectl rollout undo actually does to the ReplicaSet.

Why you need this

A release shipped, and the new version's Pods are crashing or returning errors. Fixing the code takes time, and users are hurting right now. The fastest way out: bring back the last working version. Kubernetes has kubectl rollout undo for exactly this, and it finishes in seconds, because the old version is still sitting in the cluster.

Where the history comes from

Deployment - a Kubernetes object that keeps the desired number of identical Pods running. Pod - the smallest unit of execution, a container runs inside it. ReplicaSet - an object that watches the number of Pods.

Every change to a Deployment's template (spec.template: image, env, command, probe) creates a new ReplicaSet. The old one isn't deleted right away. It stays in the cluster with replicas: 0. The set of these ReplicaSets is the history. Changing replicas (scaling) doesn't create a new revision.

The field spec.revisionHistoryLimit sets how many old ReplicaSets are kept. The default is 10. The controller deletes the extra ones, the oldest first.

To view the history:

kubectl rollout history deploy/app

deployment.apps/app
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

Each entry is a separate ReplicaSet with its own version of the Pod template. The revision number lives in the annotation deployment.kubernetes.io/revision, on both the ReplicaSet and the Deployment itself (there it's the number of the active revision). Annotation - a free-form key-value note on an object.

The CHANGE-CAUSE column is filled by the annotation kubernetes.io/change-cause on the Deployment. If nobody sets it, you get <none>. The --record flag that used to fill it is deprecated. Set the cause by hand right after an update:

kubectl annotate deploy/app kubernetes.io/change-cause="release v9" --overwrite

The annotation lands on the active revision right away and stays on the Deployment. The next update will carry it over into the new revision with the same text, so update it on every release.

To view a specific revision:

kubectl rollout history deploy/app --revision=1

The command shows the full template from that update: image, env, ports, command. You can't guess the version from the revision number, but here you can see it directly.

What kubectl rollout undo does

kubectl - the command-line tool for managing Kubernetes. Without flags, undo rolls back to the previous revision, the one right before the current one in the history:

kubectl rollout undo deploy/app

Warning: resource deployments/app was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, ...
deployment.apps/app rolled back

The warning shows up when the Deployment was created with kubectl apply. More on this below, in "Common mistakes".

With a specific revision:

kubectl rollout undo deploy/app --to-revision=1

After the rollback, the Deployment runs its usual smooth transition, just in the opposite direction. The broken ReplicaSet scales down, the target one scales up. If the template already matches the chosen revision, kubectl changes nothing and prints skipped rollback (current template already matches revision N).

Only the template is rolled back. The replica count, the strategy, and the other spec fields stay whatever they currently are.

You can preview the result without changing the cluster with the --dry-run=server flag: kubectl shows what would happen, and the object stays the same.

Deeper: what happens to the ReplicaSet on rollback

In practice, what's above is enough: undo brought back a working image, and the Pods are Running again. What follows is the mechanics under the hood, if you're curious exactly how a Deployment stores revisions.

This is the part that trips up almost everyone. It looks like a rollback creates a brand-new ReplicaSet. In reality, the Deployment reuses an existing old ReplicaSet.

What happens, step by step:

1. ReplicaSet #2 is active (revision=2, the bad version).
2. rollout undo runs.
3. The Deployment copies the template from revision=1 into its spec.template.
4. The hash (a checksum of the template) of this template matches the hash of ReplicaSet #1, which is still sitting in the cluster, scaled down to zero. The hash shows up in the ReplicaSet's name and in its pod-template-hash label. So the Deployment doesn't create a new RS, it scales #1 back up instead.
5. ReplicaSet #2 scales down to 0. ReplicaSet #1 scales up to the target replica count.
6. ReplicaSet #1 gets a new revision number: its annotation deployment.kubernetes.io/revision changes from 1 to 3, and the previous numbers pile up in the annotation deployment.kubernetes.io/revision-history ("1", and after a second such rollback, "1,3").

As a result, rollout history will show revision=2 (bad) and revision=3 (the rollback). Revision=1 disappears from the list, but the RS isn't deleted: it's the same ReplicaSet #1, just under number 3 now.

The reason lies in how a Deployment works. It keeps the current spec.template and a hash of it. A new template gives a new hash and a new ReplicaSet. A rollback brings back an old template, the hash matches an existing ReplicaSet, and that one gets reused.

You can check this:

kubectl get rs -l app=app -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.annotations.deployment\.kubernetes\.io/revision}{"\n"}{end}'

The command prints the list of ReplicaSets with their revisions. Before the rollback you'd see revisions 1 and 2, after the rollback, 2 and 3. The ReplicaSet with revision 3 has the same name that revision 1 had.

The practical takeaway: the revision number isn't the release number. After a couple of rollbacks, "the first release" might end up under number 3, 5, or 7. Before --to-revision, check a revision's content with --revision.

Can you roll back a rollback

Yes. After the rollback, revision=3 is active, and revision=2 with the bad version is still in the history. Running kubectl rollout undo again will create revision=4, copying the template from revision=2. That brings back that same bad version.

You can't roll back to a revision that's no longer in the history. That happens in two cases: a later rollback overwrote its number, or its ReplicaSet was deleted by revisionHistoryLimit. kubectl will answer:

error: unable to find specified revision 1 in history

With revisionHistoryLimit: 0, old ReplicaSets are deleted right away, and there's nowhere to roll back to at all:

error: no rollout history found for deployment "app"

Pause and rollout status

A rollback doesn't finish instantly. The Deployment runs a smooth transition:

kubectl rollout status deploy/app --timeout=120s

The command waits for it to finish. A zero exit code means the rollback went through. A nonzero one means the transition didn't finish. There are two possible cases: --timeout ran out (error: timed out waiting for the condition) or the Deployment hit progressDeadlineSeconds (error: deployment "app" exceeded its progress deadline).

Two more commands:

kubectl rollout pause deploy/app
kubectl rollout resume deploy/app

pause puts the Deployment on pause (spec.paused: true). While it's paused, template changes don't roll out: no new ReplicaSet is created, UP-TO-DATE doesn't grow, and Progressing has the reason DeploymentPaused. This is how you batch several edits into one update: pause, edit, resume. Scaling still works while paused.

A pause doesn't fix a stuck rollout, it only freezes it. And rollback is blocked while paused:

error: you cannot rollback a paused deployment; resume it first with 'kubectl rollout resume' and try again

Which revision is active right now:

kubectl get deploy app -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}'

Common mistakes

• Rolling back to a number remembered from a release. Numbers shift after every rollback, check --revision before --to-revision.
• Using plain undo when you need not the previous version, but an older one.
• Rolling back in the cluster and forgetting about the manifest. That's exactly what the Warning on undo warns about: the last-applied-configuration annotation still holds the broken version. The next kubectl apply from CI, or a GitOps sync, will roll it right back out. A cluster rollback puts out the fire, but you still have to fix it in the repository.
• Expecting a rollback to restore a ConfigMap or Secret. A revision only holds the Deployment's template. If a ConfigMap's content broke while the reference to it in the template stays the same, undo won't change anything.
• Setting pause to "stop" a bad release and forgetting about it. Every following template edit quietly piles up, and undo refuses to work until resume.

What you'll need for the tasks

• kubectl rollout history deploy/<name> and --revision=<N> to view a revision's content.
• kubectl rollout undo deploy/<name> and --to-revision=<N>.
• kubectl rollout status, kubectl rollout pause, kubectl rollout resume.
• kubectl get rs -l <selector> and the annotation deployment.kubernetes.io/revision.
• kubectl describe deploy <name> (the Conditions block) and kubectl describe pod <name> (the Events block).

The labs are in the labs/ folder: each one brings up its own kind cluster on your computer, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 3 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-rollback     # set up the starting state
./run.sh verify k8s-rollback    # check the current step

In short

• The revision history is old ReplicaSets, scaled down to replicas: 0.
• revisionHistoryLimit (default 10) sets how far back you can roll.
• undo doesn't create a new ReplicaSet, it brings the old one back up by its matching template hash.
• Revision numbers only go up: rolling back to 1 creates entry 3, and entry 1 disappears.
• A rollback restores only the template, not replicas and not a ConfigMap.
• rollout status --timeout returns a nonzero code if the rollback hasn't stabilized. Rollback is blocked while paused.