kubectl edit and immutable Pod fields

What you can change on a live Pod, what you can't, and how to work around the limit.

Why this matters

A Pod is already running, and something needs fixing: the image, a label, the CPU limit. The first thought is to open kubectl edit and fix the field. For the image, that works. For nodeSelector or an environment variable, the API server returns Forbidden, and the edit isn't saved. This lesson is about where that line runs, and what to do on the other side of it.

What kubectl edit does

A Pod is the smallest unit Kubernetes runs. One or more containers run inside a Pod. spec is the part of the manifest that describes the object's desired state.

kubectl edit <kind>/<name> pulls the current object from the API, opens it in a temporary file in an editor, and sends your changes back on exit. If you changed nothing, kubectl prints Edit cancelled, no changes made. and sends nothing. If you did change something, the change goes to the API server and runs through validation.

In spirit, it's three steps rolled into one command:

kubectl get pod web -o yaml > /tmp/pod.yaml
vi /tmp/pod.yaml
kubectl apply -f /tmp/pod.yaml

The difference is in the details: edit sends only the changed fields (a patch) and, without the --save-config flag, doesn't add the last-applied-configuration annotation that apply sets.

The editor comes from the KUBE_EDITOR variable, then from EDITOR. If both are empty, vi opens on Linux and macOS, notepad on Windows. You can switch editors like this:

export KUBE_EDITOR=nano
kubectl edit pod web

The map: what changes, what doesn't

First the big picture, then we'll go through it point by point.

Changes with plain edit	Only through a separate path	No, you have to recreate
Container and init container image	resources (the resize subresource)	nodeSelector
labels, annotations		serviceAccountName
activeDeadlineSeconds (decrease only)		env, command, ports
tolerations (additions only)		volumes, securityContext

What you can change on a running Pod

Once a Pod is created, the API server only allows changing a narrow set of spec fields. The main exception is the container image.

spec.containers[*].image can be changed. kubelet pulls the new image and restarts the container: the RESTARTS counter goes up by one. The Pod itself stays, its UID is unchanged, no new Pod is created.

kubectl set image pod/web app=nginx:1.27.3-alpine

kubectl set image changes the image directly, without opening an editor. The format: pod/<name> <container>=<image>. Here the container name is app, not the image name. Get it wrong, and kubectl replies error: unable to find container named "nginx". You can look up container names like this:

kubectl get pod web -o jsonpath='{.spec.containers[*].name}'

The rest of what's allowed on a live Pod:

• metadata.labels and metadata.annotations. These aren't spec, they're changed with kubectl label and kubectl annotate.
• spec.initContainers[*].image, the init container's image.
• spec.activeDeadlineSeconds: set it if it wasn't there, or decrease it. You can't increase it, the reply is must be less than or equal to previous value.
• spec.tolerations: only adding new ones.

What you can't change

The rest of the spec fields are immutable after the Pod is created. If you try to fix them, the validator returns an error. The most common ones:

• spec.containers[*].resources: not through a plain edit, but there's an official path through the resize subresource (see below).
• spec.containers[*].env, command, args, ports: no.
• spec.serviceAccountName: no.
• spec.nodeSelector: no, even while the Pod is still sitting in Pending.
• spec.volumes, spec.securityContext: no.

The error looks like this. The full text also lists what's allowed:

The Pod "web" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`,`spec.initContainers[*].image`,`spec.activeDeadlineSeconds`,`spec.tolerations` (only additions to existing tolerations),`spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)

Below it, kubectl prints a diff: which field you tried to change. If the edit went through kubectl edit, your version isn't lost, kubectl saves it to a temporary file:

error: pods "web" is invalid
A copy of your changes has been stored to "/tmp/kubectl-edit-1456954052.yaml"
error: Edit cancelled, no valid changes were saved.

This is intentional. kubelet has already started the container with specific settings. Changing them on the fly would leave the manifest saying one thing while the container does another.

Resources change on the fly: the resize subresource

CPU and memory on a live Pod can be changed without recreating it. This is called in-place resize. The feature has been on by default since Kubernetes 1.33 and became stable in 1.35. It works through a separate resize subresource, not through a regular Pod update. --subresource resize needs kubectl 1.32 or newer.

The plain route is still forbidden:

kubectl patch pod web --type=json \
  -p '[{"op":"replace","path":"/spec/containers/0/resources/limits/cpu","value":"200m"}]'

The Pod "web" is invalid: spec: Forbidden: pod updates may not change fields other than ...

And this is how you change the limit on a running Pod:

kubectl patch pod web --subresource resize --type=json \
  -p '[{"op":"replace","path":"/spec/containers/0/resources/limits/cpu","value":"200m"}]'

pod/web patched

spec holds what you asked for. What kubelet has actually applied to the container shows up in status. Right after the patch these values can differ, they converge within a couple of seconds:

kubectl get pod web -o jsonpath='{.spec.containers[0].resources}'
kubectl get pod web -o jsonpath='{.status.containerStatuses[0].resources}'

If the change can't be applied right away, Kubernetes sets the Pod condition PodResizePending. Reason Infeasible means the node doesn't have that much capacity at all, Deferred means there's no room right now, but there might be later. While kubelet is applying the change, the condition PodResizeInProgress is set. For example, requesting 64 CPU on a node with 4 cores:

PodResizePending=True Infeasible Node didn't have enough capacity: cpu, requested: 64000, capacity: 4000

The rule "a Pod is almost immutable" still holds, resize doesn't change that. It only covers CPU and memory. You still can't change nodeSelector, volumes, or serviceAccountName on a live Pod.

How to change what's immutable

We covered resources above, they change through resize. For the rest of the immutable fields, there's one path: recreate the Pod. That's what kubectl replace --force is for:

kubectl get pod web -o yaml > /tmp/pod.yaml
# fix the field you need in the file
kubectl replace --force -f /tmp/pod.yaml

pod "web" deleted from default namespace
pod/web replaced

--force means: delete the old Pod and create a new one from the file. It's shorthand for two commands:

kubectl delete pod web
kubectl create -f /tmp/pod.yaml

Without --force, replace tries to update the existing object and runs into the same Forbidden.

The Pod's name stays the same, but this is a completely new object. It gets a new UID (the object's internal identifier) and a new IP, and restart counters start back at zero. The Pod isn't updated, it's deleted and created again. Everything that lived in its temporary volumes is gone along with the old Pod.

Why production doesn't run Pods without a controller

A Deployment is an object that manages a set of identical Pods and knows how to update them without downtime. When you change the template in a Deployment, it doesn't edit existing Pods, it creates new ones to replace them. That's why a Deployment lets you change image, resources, and nodeSelector alike:

kubectl set image deploy/app app=nginx:1.27.3-alpine

Here app is also the container name in the Deployment's template. For a Deployment created with kubectl create deployment --image=nginx:..., the container is named nginx.

In production, apps run under a Deployment so updates go through a smooth replacement, not manual delete and recreate.

Common mistakes

• kubectl edit opened, the edit to env or nodeSelector was saved, but the reply is Forbidden. The field is immutable, the edit didn't apply. Your version sits in the temporary file named in the message, and you can hand it to kubectl replace --force -f.
• kubectl set image given the image name instead of the container name: unable to find container named .... Check names with the jsonpath {.spec.containers[*].name}.
• kubectl patch on resources without --subresource resize: the same Forbidden.
• Expecting resize to change something other than CPU and memory. It won't.
• replace --force on a Pod with data in emptyDir: the data is gone, because it's a new Pod.
• The edit goes to the wrong cluster. Before making changes, check kubectl config current-context.

What you'll need for the tasks

• kubectl get pod <name> -o jsonpath='{.spec.containers[*].name}', to find out the container's name.
• kubectl set image pod/<name> <container>=<image> and kubectl label pod.
• kubectl patch pod <name> --subresource resize ... for CPU and memory.
• kubectl describe pod: the reason for Pending is in the Events block.
• kubectl get pod <name> -o yaml > file, edit the file, then kubectl replace --force -f file.
• The Pod's UID: kubectl get pod <name> -o jsonpath='{.metadata.uid}'. If it changed, the Pod was recreated.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your machine, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-edit-pod     # set up the starting state
./run.sh verify k8s-edit-pod    # check the current step

In short

• In spirit, kubectl edit is three steps in one: get the YAML, edit it, send the change.
• A live Pod lets you change containers' and init containers' image, labels, annotations, decreasing activeDeadlineSeconds, and adding tolerations.
• nodeSelector, env, volumes, serviceAccountName, securityContext are immutable, the error looks like spec: Forbidden.
• CPU and memory change through kubectl patch --subresource resize.
• For the rest of the immutable fields you need kubectl replace --force, which is delete + create with a new UID and IP.
• A Deployment has none of these limits: it replaces Pods with new ones from the template.