Helm: upgrade and rollback

Why

You don't reinstall an installed release. You change it with the helm upgrade command, and Helm keeps a history for it: every change becomes a new revision. If the new version breaks, rolling back takes one command, no manual rebuilding of manifests. In CI this is Helm's main workflow: deploy, check, bring back if needed.

Concepts

Revision: the number of a release's state. An install gives revision 1, every upgrade and every rollback adds the next one.

Every revision has a status:

• deployed: the release's current working state;
• superseded: a revision that a newer successful one replaced;
• failed: the update or rollback didn't succeed;
• pending-install, pending-upgrade, pending-rollback: an operation is running right now, or it was interrupted halfway through.

Helm stores each revision's state in a separate Secret in the release's namespace, named sh.helm.release.v1.<release>.v<revision>, for example sh.helm.release.v1.web.v2.

How it works

Upgrade

helm upgrade web ./myapp -n demo -f values/prod.yaml --set-string image.tag=1.27.3-alpine

Helm renders the templates again with the new values and applies the result to the cluster. Release web gets revision 2, the previous one stays in the history with status superseded.

How exactly the changes get applied depends on the version. Helm 3 builds a patch from three sources: the previous revision's manifest, the new manifest, and the live object in the cluster. Helm 4 uses server-side apply for new releases, and keeps the old method for releases installed the old way (the --server-side flag defaults to auto). For the commands in this lesson it makes no difference, it only shows up in the wording of error messages.

In practice people almost always add the --install flag:

helm upgrade --install web ./myapp -n demo -f values/prod.yaml

Then the first run installs the release, and later runs update it. The same command works on both a clean cluster and an existing one, which is why CI pipelines like it.

Upgrade and previous values

This is the lesson's main trap. If you pass even one -f or --set to upgrade, Helm rebuilds the values from scratch: the chart's defaults plus whatever this command passed. Anything set at the previous install and not passed now disappears.

Example. The release was installed with -f values/prod.yaml, where replicaCount: 3 and config.env: prod. Then only the version got updated:

helm upgrade web ./myapp -n demo --set-string config.version=2

helm get values web -n demo

USER-SUPPLIED VALUES:
config:
  version: "2"

Replicas are back to 1, config.env fell back to the chart's default. But if you run helm upgrade with no -f and no --set at all, Helm takes the previous revision's values whole. So the mistake doesn't always show up, only when you pass something.

There are two fixes:

• pass the same values files in every command (-f values/prod.yaml --set-string config.version=2). That way the values always live in git, and the command is easy to repeat;
• --reuse-values: take the previous revision's values and layer the new ones on top. Downside: if the new chart version added a key to its own values.yaml, --reuse-values won't pick it up. For that case there's --reset-then-reuse-values: first the new chart's defaults, then the previous values, then the new ones.

Revision history

helm history web -n demo

REVISION  UPDATED                   STATUS      CHART      APP VERSION    DESCRIPTION
1         Mon Oct 12 17:58:47 2026  superseded  web-0.1.0  1.27.3-alpine  Install complete
2         Mon Oct 12 17:58:48 2026  deployed    web-0.1.0  1.27.3-alpine  Upgrade complete
3         Mon Oct 12 17:58:48 2026  failed      web-0.1.0  1.27.3-alpine  Upgrade "web" failed: ...

You see the number, date, status, chart version, and description: install, upgrade, or rollback. A failed update (revision 3) doesn't touch revision 2's status: it stays deployed.

A specific revision's values come from the same Secret:

helm get values web -n demo --revision 1

helm get manifest web -n demo --revision 1 works the same way: it shows which manifests were applied in that revision.

Rollback

helm rollback web 1 -n demo

The release goes back to revision 1's state: the same manifests and the same values.

A rollback doesn't erase the history. It adds a new revision with a copy of the old state and description Rollback to 1. The logic is the same as kubectl rollout undo, just at the level of the whole release instead of a single Deployment.

rev 1  install    deployed -> superseded
rev 2  upgrade    broken
rev 3  rollback   copy of rev 1, description "Rollback to 1"

Without a number, helm rollback web rolls back to the revision numbered one less than the last one. That's not necessarily the last working one: if the last one was a failed attempt to fix something, a rollback with no number brings back the very revision that attempt was trying to fix. Better to pick the number from helm history and state it explicitly.

How many revisions to keep

Every revision takes up a separate Secret. By default helm upgrade keeps the last 10 and deletes older ones. The limit changes with a flag:

helm upgrade web ./myapp -n demo -f values/prod.yaml --history-max 3

A value of 0 removes the limit. On a release that updates often, that turns into hundreds of Secrets, so the limit is usually left in place. You can only roll back to a revision that's still kept.

Waiting for the result and rolling back automatically

By default helm upgrade finishes as soon as the API server accepts the manifests. Helm doesn't check whether the Pods came up: a release with a broken image tag still gets status deployed.

--wait can wait for readiness, and automatic rollback turns on with a flag that's named differently across the two versions:

# Helm 4
helm upgrade web ./myapp -n demo -f values/prod.yaml --rollback-on-failure --timeout 5m
# Helm 3
helm upgrade web ./myapp -n demo -f values/prod.yaml --atomic --timeout 5m

The flag turns on waiting for readiness, and if something didn't come up within --timeout, it rolls the release back to the previous working revision. Both entries stay in the history: the failed update with status failed, and a revision Rollback to <N>. In Helm 4, --atomic still works, but it prints Flag --atomic has been deprecated, use --rollback-on-failure instead.

For pipelines this is the most useful flag: a failed rollout doesn't leave the release in a half-broken state.

Separately there's --wait (wait without rolling back: the release gets failed, and the new manifests stay in the cluster) and --cleanup-on-fail (delete the objects a failed update created). In Helm 4, --wait takes a strategy: plain --wait means watcher, and without the flag hookOnly applies, meaning Helm waits only for hooks.

Common mistakes

An interrupted update leaves the release in status pending-upgrade. This happens if the helm process got killed mid---wait, or the CI runner crashed. The next command refuses to run:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

The way out: find the last working revision with helm history and roll back to it with helm rollback.

Immutable fields don't survive an upgrade. Changing a Deployment's spec.selector or a Secret's type field gets an error from the API server like spec.selector: Invalid value: ...: field is immutable. Objects like that have to be deleted and created again.

Rollback brings back the manifests, but not the data. A database migration the new app version ran stays applied, and the old code might not start up with it. Backward compatibility of the database schema stays the application's job, not Helm's.

A revision with status failed still takes up a number. The history can have several failed revisions in a row, and you need to roll back to the last one with status deployed or superseded that actually worked, not just the previous number.

An upgrade with only some of the flags loses the rest of the values. If CI passes only --set image.tag=..., everything from the prod file disappears on the very first rollout. Check helm get values <release> before and after.

What you'll need for the tasks

• helm history <release> -n <ns>: revision numbers, statuses, descriptions.
• helm get values and helm get manifest with --revision <N>: what was in a specific revision.
• helm rollback <release> <N> -n <ns> with an explicit number.
• helm upgrade --install with -f, --set-string, --reuse-values.
• --rollback-on-failure (Helm 4) or --atomic (Helm 3) with --timeout.
• For the mini-project, also Kustomize from the start of the module: overlay, configMapGenerator, secretGenerator, kubectl apply -k.
• In every lab: kubectl config current-context before any changes.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-helm-upgrade-rollback     # set up the starting state
./run.sh verify k8s-helm-upgrade-rollback    # check the current step

In short

• helm upgrade --install installs the release if it's missing and updates it if it exists.
• Every change creates a revision, its state lives in Secret sh.helm.release.v1.<release>.v<N>.
• helm history shows the revisions and statuses, helm rollback <release> <N> brings back revision N's state.
• A rollback doesn't erase the history, it adds one more revision on top. Without a number it goes to the previous number, not the last working revision.
• --history-max limits how many revisions are kept, 10 by default.
• --rollback-on-failure (--atomic in Helm 3) waits for readiness and rolls back a failed rollout on its own.
• An upgrade with even one -f or --set forgets the previous values. Pass the same files, or use --reuse-values.