Rolling update: updating without downtime

How a Deployment moves Pods from the old ReplicaSet to the new one, what maxSurge and maxUnavailable do, and where revisions come from.

Why this matters

Every release in Kubernetes means changing a Deployment's Pod template. If you don't understand how the transition works, it's easy to end up with downtime for no reason, or an update that quietly gets stuck halfway. The transition settings are short, but they decide how many Pods are running at every moment of the release.

Concepts

Deployment is a Kubernetes object that keeps the wanted number of identical Pods running. Pod is the smallest unit Kubernetes runs, with a container working inside it. ReplicaSet is an object that makes sure the number of Pods matches the requested count.

Revision is a saved version of the Pod template at the moment of a change. Each revision has its own ReplicaSet behind it.

Annotation is an arbitrary key-value tag on an object.

What Deployment does during an update

A Deployment stores the Pod template in the spec.template field. The template changes: a new image, new env, new resources. After that the Deployment creates a new ReplicaSet and starts a smooth transition.

The old ReplicaSet is gradually scaled down to zero Pods. The new one grows to the needed replica count. This smooth transition is called a rolling update.

RS old (v1)  ●●● → ···  3 → 0 Pods
RS new (v2)  ··· → ●●●  0 → 3 Pods

The transition's behavior is controlled by the spec.strategy block:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%

maxSurge is how many Pods beyond replicas are allowed to come up during the transition. maxUnavailable is how many of the wanted Pods are allowed to be not in a working state during the transition. At zero, new Pods come up first, and only then are the old ones killed.

Both fields accept a number or a percentage. A percentage of replicas rounds up for maxSurge and down for maxUnavailable. By default both are 25%. At replicas: 3 this gives one extra Pod and zero unavailable: Kubernetes first brings up a fourth Pod of the new image and kills the old one only once the new one is Ready.

On a tight cluster, people set maxSurge: 0 to avoid creating extra Pods. If availability matters for the app, they set maxUnavailable: 0, so new Pods come up before the old ones are turned off. You can't set both to zero at once: the transition has nothing to move with, and the API answers with an error:

spec.strategy.rollingUpdate.maxUnavailable: Invalid value: 0: may not be 0 when `maxSurge` is 0

The second strategy is Recreate. It kills all the old Pods first, then brings up all the new ones. This guarantees a short downtime. It's needed for apps that can't run two versions at the same time.

kubectl set image

kubectl is the command-line tool for managing Kubernetes. You can change an image by editing the YAML and running kubectl apply. For quick edits and in scripts, this is done with one command:

kubectl set image deploy/app app=nginx:1.27.3-alpine

To the left of = is the container name in the Deployment, to the right is the new image. The container name doesn't have to match the Deployment name: for example, kubectl create deployment app --image=nginx:1.27.3-alpine creates a container named nginx. To see the container names:

kubectl get deploy app -o jsonpath='{.spec.template.spec.containers[*].name}'

The set image command edits spec.template, and the rolling update starts right away. In production the image is usually changed in the manifest or chart that CI applies, otherwise the next deploy from Git will bring back the old value.

Wait for the transition to finish:

kubectl rollout status deploy/app

The command returns exit code zero once all replicas become Ready on the new template and the old Pods are gone.

If you point at a tag that doesn't exist in the registry, kubelet won't be able to pull the image. The new Pod gets stuck in status ErrImagePull, then ImagePullBackOff. With maxUnavailable: 0 the old Pods keep serving traffic, so there's no downtime, but the transition won't finish until the image is fixed or you roll back. rollout status waits as long as set in spec.progressDeadlineSeconds (600 seconds by default), then exits with code 1:

error: deployment "app" exceeded its progress deadline

To record the reason for an update, add an annotation:

kubectl annotate deploy/app kubernetes.io/change-cause="upgrade to nginx 1.27.3"

This annotation ends up in rollout history, in the CHANGE-CAUSE column.

rollout history

The command kubectl rollout history deploy/app shows all revisions:

deployment.apps/app
REVISION  CHANGE-CAUSE
1         <none>
2         upgrade to nginx 1.27.3

The Deployment itself and each of its ReplicaSets keep the revision number in the annotation deployment.kubernetes.io/revision.

How many old ReplicaSets stay in the history is set by the spec.revisionHistoryLimit field. By default it's 10. At 0 the history is wiped, and there's nowhere to roll back to.

See the full template of a specific revision:

kubectl rollout history deploy/app --revision=2

The command shows the image, env and ports for that revision without rolling back. If that revision doesn't exist, it answers with error: unable to find the specified revision.

Rollback in brief

If the new version turns out broken, you unwind the transition:

kubectl rollout undo deploy/app
kubectl rollout undo deploy/app --to-revision=1

Without arguments the command brings back the previous revision. After a rollback the revision number doesn't repeat. You rolled back from revision=2 to revision=1, and the history gets a revision=3 with the same template that revision=1 had, while revision=1 drops out of the list. The number always grows. What happens to the ReplicaSets under the hood when this happens is covered in the next lesson.

Useful commands

# What is there right now
kubectl get deploy app -o wide
kubectl get rs -l app=app

# Current image
kubectl get deploy app -o jsonpath='{.spec.template.spec.containers[0].image}'

# Current strategy
kubectl get deploy app -o jsonpath='{.spec.strategy}'

# History
kubectl rollout history deploy/app

Common mistakes

• Writing the Deployment name instead of the container name in set image. kubectl answers unable to find container named ..., and the template doesn't change.
• Setting maxSurge: 0 and maxUnavailable: 0 together. The API won't accept such a Deployment.
• Expecting the new ReplicaSet to always come up. If the namespace has a ResourceQuota on the number of Pods or on memory, the extra Pod from maxSurge might not fit into it. Then the ReplicaSet can't create the Pod, and the reason is visible only in its events and in kubectl describe deploy.
• Assuming that rollout status exiting with code 0 guarantees a working app. It only checks Ready, and Ready depends on readinessProbe. Without a readiness check, a Pod is considered Ready right after the container starts.
• Setting revisionHistoryLimit: 0 to save space, and losing the ability to roll back.

What you'll need for the tasks

• kubectl get deploy, kubectl get rs, kubectl get pods and the UP-TO-DATE column.
• kubectl get deploy <name> -o jsonpath='{.spec.strategy}' and a way to change the strategy: kubectl edit deploy <name> or kubectl patch.
• kubectl set image deploy/<name> <container>=<image>.
• kubectl annotate with the key kubernetes.io/change-cause.
• kubectl rollout status, kubectl rollout history.
• kubectl describe deploy <name> and kubectl describe rs <name>: the Conditions and Events blocks.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your computer, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 3 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-rolling-update     # set up the starting state
./run.sh verify k8s-rolling-update    # check the current step

In short

• Changing spec.template creates a new ReplicaSet, and the old one shrinks to zero.
• maxSurge allows extra Pods, maxUnavailable allows missing ones. Percentages for the first round up, for the second round down.
• maxUnavailable: 0 keeps availability, maxSurge: 0 saves cluster resources. Setting both to zero is not allowed.
• Recreate kills everything at once, good for apps without cross-version compatibility.
• A broken image tag doesn't take down production, but it hangs the rollout in ImagePullBackOff. rollout status gives up after progressDeadlineSeconds.
• change-cause via annotation keeps the revision history readable.