Immutable Secret: blocking changes to data

Why immutable

Secret and ConfigMap have an immutable field. It's been stable since Kubernetes 1.21. Set immutable: true and the apiserver refuses to change data, stringData, and binaryData. The only way to change the data is to delete the object and create it again.

Why you'd want this:

1. Protection from accidental edits. A config value or key can't "drift" through a stray kubectl edit secret prod.
2. Less load on kube-apiserver. For every mutable Secret and ConfigMap mounted into a Pod, kubelet watches it so it can update the files in the volume. For an immutable object the watch is closed: the data won't change, so there's nothing to watch for. On a cluster with thousands of Pods, that's a real saving in API requests.
3. Immutable deploys. CI generates a unique name (app-config-abc123), the deploy references it, and after a rotation a new app-config-def456 is created. The old one is deleted once nothing references it anymore.

The manifest

apiVersion: v1
kind: Secret
metadata:
  name: db-password
type: Opaque
immutable: true
data:
  password: c2VjcmV0MTIz

data is always base64 (here c2VjcmV0MTIz is secret123). stringData is written as plain text, and the apiserver encodes it on save. When you read the object back, stringData is gone and the value lives in data.

You don't have to set immutable at creation time. You can turn it on later on an existing Secret or ConfigMap, and the object isn't recreated in the process. But you can't turn it back off: the transition true → false is forbidden. To change the data, you recreate the object.

What's forbidden

Any attempt to change the data ends with the same apiserver error:

kubectl patch secret db-password \
  --type=merge -p '{"stringData":{"password":"new"}}'

The Secret "db-password" is invalid: data: Forbidden: field is immutable when `immutable` is set

stringData turned into data in the error: the apiserver moves it into data first, then validates. kubectl apply of the same file with a different password gets the same answer. kubectl edit won't save the change either: it prints error: secrets "db-password" is invalid and Edit cancelled, no valid changes were saved.

Forbidden: changing data, stringData, binaryData, and switching immutable: true → false. For a ConfigMap the error is the same, just with its name and field (data or binaryData).

Allowed: editing metadata.labels and metadata.annotations, for example kubectl label secret db-password team=platform. You can also still delete the object and create it again, that's two separate API requests.

Updating an immutable Secret

The main pattern is rotation through a new name. The old Secret stays untouched, a new one is created next to it:

apiVersion: v1
kind: Secret
metadata:
  name: db-password-v2
immutable: true
type: Opaque
data:
  password: bmV3LXNlY3JldA==

In the Deployment you update the reference, it used to be db-password-v1:

envFrom:
  - secretRef:
      name: db-password-v2

Editing the Pod template triggers a rollout: the Deployment creates new Pods, and they read db-password-v2 already. The old Secret is deleted last, once no replica is looking at it anymore. Delete it too early and Pods that get recreated with the old template won't start: status CreateContainerConfigError and event secret "db-password-v1" not found.

A cruder alternative: kubectl delete secret followed right away by kubectl create secret with the same name. Running Pods won't see the new data. Environment variables are read once, at container start, and kubelet doesn't update the files in the volume for an immutable object, because it isn't watching it. The new values only show up in Pods that get recreated. In the gap between delete and create, any new Pod crashes with CreateContainerConfigError. That's why rotating through a new name and a rollout is the cleaner way.

Common mistakes

• Setting immutable: true in production and forgetting that rotation now goes through a new name. Any data update after that gets a Forbidden.
• Rebuilding a Secret with kubectl create secret ... --dry-run=client -o yaml | kubectl apply -f -. The generated YAML has no immutable, and the apiserver refuses twice: once for data, once for trying to drop immutable.
• Wanting to turn the flag off. You can't, only delete and recreate.
• Deleting the old Secret before the rollout finishes. New Pods with the old template, and Pods on rollback, won't find it and won't start.
• Recreating a Secret under the same name and expecting running Pods to pick up the new data. They won't, not through env, not through a volume.

What you'll need for the tasks

• Lab 1 (labs/01-lock-config): turn on immutable on an already existing Secret and ConfigMap without recreating them, see the apiserver reject a data edit, and put a label on the locked Secret (kubectl label).
• Lab 2 (labs/02-rotate-key): rotation through a new name. A new immutable Secret (stringData or kubectl create secret generic), updating the reference in envFrom on a Deployment (kubectl edit deployment), kubectl rollout status, deleting the old Secret.
• Commands: kubectl get ... -o yaml, kubectl get ... -o jsonpath='{.immutable}', kubectl patch, kubectl label, kubectl logs, kubectl exec ... printenv.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-immutable-secret     # set up the starting state
./run.sh verify k8s-immutable-secret    # check the current step

In short

• immutable: true forbids changing data, stringData, and binaryData on a Secret or ConfigMap.
• You can turn the flag on for an existing object, but you can't turn it back off.
• Labels and annotations can still be edited.
• Kubelet doesn't keep a watch on an object like that, which saves API requests on large clusters. The flip side: files in a volume won't update on their own.
• Updates happen through rotation: a new name, updating the reference in the Deployment, a rollout, deleting the old object.