Manual scheduling: putting a Pod on a node past the scheduler

What happens when you write a node's name into a manifest by hand, why a Pod with a typo in that name quietly disappears, and how kubelet runs the control plane itself, from files.

Why this matters

Normally you do not choose a node for a Pod, the scheduler does. But you can fill in the field spec.nodeName yourself, and then the scheduler takes no part in the choice. That is how diagnostic Pods get pinned "to a specific node". Static pods work the same way, and the control plane in kubeadm clusters runs on them. It is also how manifests copied from a different cluster break.

How a Pod normally ends up on a node

The scheduler (kube-scheduler) watches for Pods with an empty spec.nodeName. It drops nodes that do not fit by taints, affinity, and resources, scores the rest, and picks the best one. It writes the result into the field spec.nodeName, say node-3. Kubelet on that node sees a Pod with its own name and starts the containers.

Manual scheduling

Manual scheduling means you fill in spec.nodeName yourself. The scheduler only picks up Pods with an empty field, so it skips a Pod like this. The kubelet on the named node treats the Pod as its own right away.

regular   Pod → scheduler → kubelet
nodeName  Pod ──────────────→ kubelet

Minimal manifest:

apiVersion: v1
kind: Pod
metadata:
  name: pinned-pod
spec:
  nodeName: worker-2
  containers:
    - name: nginx
      image: nginx:1.27.3-alpine

The Kubernetes docs explicitly say not to use this field to express "I want this on that node" (kubectl explain pod.spec.nodeName). That is what nodeSelector and nodeAffinity are for, covered in the next lessons.

Which checks disappear, and which stay

The scheduler is out of the chain, so its filters are too. The most noticeable one to go is the taints filter. Take a node with the taint dedicated=reports:NoSchedule: a regular Pod on it stays Pending with an event:

Warning  FailedScheduling  default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint(s). ...

A Pod with nodeName pointing at the same node started and became Running, even with no toleration. A taint with effect NoSchedule is only checked by the scheduler.

But kubelet also checks a Pod before starting it. If the node does not have enough resources for the requests, or the Pod does not match the nodeSelector, kubelet rejects it. The Pod goes straight to Failed, and the STATUS column shows the reason:

kubectl get pods

NAME     READY   STATUS         RESTARTS   AGE
badsel   0/1     NodeAffinity   0          33s
bigcpu   0/1     OutOfcpu       0          34s
bigmem   0/1     OutOfmemory    0          33s

Kubelet's message in kubectl describe pod:

Pod was rejected: Node didn't have enough resource: cpu, requested: 100000, used: 950, capacity: 4000

A Pod like this does not restart and does not move to another node. You have to delete it and create it again with the right values.

A taint with effect NoExecute also still applies: a Pod with nodeName gets evicted from the node, like any other Pod without the right toleration.

Can you tell the difference

A Pod the scheduler placed has an event from default-scheduler:

Normal  Scheduled  default-scheduler  Successfully assigned default/plain to ...

A Pod with nodeName has no such event, its first events come straight from kubelet (Pulled, Created, Started). In a single-node cluster, that is the only way to tell one Pod from another, since by placement they will end up on the same node either way.

A typo in the node name

If no node with that name exists, no one picks up the Pod. The scheduler does not look at it, because the field is already filled in. The Pod sits in Pending, the NODE column shows a name that does not exist, and there are no events at all.

It does not sit there for long. The PodGC controller in kube-controller-manager treats a Pod like this as orphaned and deletes it. A Pod with nodeName: worker-2 lives like this for about 75 seconds and vanishes without a single event. The kube-controller-manager log is left with:

"PodGC is force deleting Pod" pod="default/ghost3"
"Forced deletion of orphaned Pod succeeded" pod="default/ghost3"

From the outside this looks like "the Pod got created and disappeared". The first thing to do in a situation like this: check spec.nodeName in the manifest and compare it with kubectl get nodes.

You cannot fix nodeName on a live Pod. The API responds:

The Pod "pinned" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, ...

The Pod gets deleted and recreated with the right name.

nodeName in a Deployment

The field lives in the Pod's own spec. In a Deployment that is spec.template.spec.nodeName, and then it applies to every replica. The scheduler skips all of them, and all the replicas land on one node. If that node goes down, every replica goes down with it.

The right way to pin a Deployment to a group of nodes is nodeSelector or nodeAffinity.

Static pods

A static pod is a Pod that kubelet launches by itself from a YAML file on the node's disk, without the API server involved. Kubelet reads the directory named in the staticPodPath field of its own config. In kubeadm clusters and in kind that is /etc/kubernetes/manifests. The old kubelet flag --pod-manifest-path does the same thing, but it is deprecated: the setting moved into the config file.

A kind node is a Docker container, so you can look inside it with docker exec. The container name matches the node name from kubectl get nodes:

docker exec kind-control-plane grep staticPodPath /var/lib/kubelet/config.yaml

staticPodPath: /etc/kubernetes/manifests

docker exec kind-control-plane ls /etc/kubernetes/manifests

etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml

This is how kubeadm solves the chicken-and-egg problem. The API server itself runs as a Pod, and you cannot bring it up through the API server. Kubelet launches it from a file, no API server needed for that.

Mirror pod

For a static pod to show up in kubectl, kubelet creates its reflection in the API, a mirror pod. The node name gets appended to the Pod's name:

kubectl -n kube-system get pods

NAME                                       READY   STATUS    RESTARTS   AGE
etcd-kind-control-plane                    1/1     Running   0          16m
kube-apiserver-kind-control-plane          1/1     Running   0          16m
kube-scheduler-kind-control-plane          1/1     Running   0          16m
...

You can tell a mirror pod from a regular one by the annotation kubernetes.io/config.source (a static pod has the value file) and by its owner: ownerReferences points to a Node object.

kubectl -n kube-system get pod kube-scheduler-kind-control-plane \
  -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.source}{" "}{.metadata.ownerReferences[0].kind}{"\n"}'

file Node

Only the file controls a static pod. kubectl delete removes the mirror pod, but the containers keep running, and kubelet recreates the reflection a few seconds later. To stop a static pod, remove the file from the manifests directory. Kubelet rereads the directory periodically (the field fileCheckFrequency, 20 seconds by default), so the Pod does not disappear instantly: the mirror pod vanishes about 45 seconds after the file was removed.

Related commands

kubectl get nodes
kubectl get pods -o wide                     # NODE column for every Pod
kubectl get pod <name> -o jsonpath='{.spec.nodeName}'
kubectl describe pod <name>                  # events: was there a Scheduled
kubectl describe node <name>                 # taints, resources, Pods on the node
kubectl explain pod.spec.nodeName
docker exec <kind-node-name> ls /etc/kubernetes/manifests

Common mistakes

• A manifest with nodeName was moved to a different cluster. The nodes there are named differently, the Pod sits in Pending with no events for a minute and disappears. Check the NODE column in kubectl get pods -o wide and compare it with kubectl get nodes.
• Assuming nodeName bypasses every check. It bypasses NoSchedule taints, but kubelet still checks resources and nodeSelector itself. The Pod goes to Failed with status OutOfcpu, OutOfmemory, or NodeAffinity.
• nodeName in a Deployment template. All replicas on one node, no fault tolerance.
• Deleting a static pod with kubectl delete. The mirror pod comes back. You have to delete the file on the node.
• A static pod did not show up in kubectl get pods. Reasons: the file is not in the directory from staticPodPath, there is an error in the YAML, or it names a namespace that does not exist in the cluster. In the last case, the container on the node is running, but the API will not accept the mirror pod. Check kubelet's logs on the node: journalctl -u kubelet (in kind, via docker exec <node> journalctl -u kubelet).

What you'll need for the tasks

• kubectl get nodes and kubectl get pods -o wide: compare the NODE column with the actual node names.
• kubectl describe pod <name>: whether there are events, and from whom.
• Recreating a Pod: kubectl delete pod <name> and kubectl apply -f <file>, since nodeName cannot be changed on a live Pod.
• docker exec <node> ... and docker cp <file> <node>:<path>: look inside the kind node and put a file there.
• The annotation kubernetes.io/config.source and the node-name suffix on a mirror pod.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your machine, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-manual-scheduling     # set up the starting state
./run.sh verify k8s-manual-scheduling    # check the current step

In short

• A filled-in spec.nodeName takes the scheduler out of the chain: the Pod is picked up by the kubelet on the named node.
• NoSchedule taints have no effect here. Kubelet still checks resources and nodeSelector itself, and rejects the Pod with status OutOfcpu, OutOfmemory, or NodeAffinity.
• A Pod pinned to a node that does not exist sits in Pending with no events, and PodGC deletes it a minute or two later.
• nodeName cannot be changed on a live Pod, the Pod gets recreated.
• Kubelet launches static pods from files in staticPodPath, and the API shows their mirror pods with the node name at the end. The only way to stop a static pod is to delete the file.