DaemonSet: one Pod per node

A controller for agents that must run on every machine: how it counts Pods, and how nodeSelector and taints fit in.

Why you need a DaemonSet

DaemonSet is a controller that keeps one Pod on every node in the cluster that qualifies. The number of copies isn't something a person writes. The controller derives it from the set of nodes: five nodes, five Pods; fifty nodes, fifty Pods.

The set of nodes changes, and DaemonSet tracks that continuously. Add a machine to the cluster, and a Pod appears on it by itself, no command needed. Remove a node from the cluster, and its Pod gets deleted along with it.

This is how you set up everything that has to run on every machine: a log collector, a metrics exporter like node-exporter, a CNI network plugin, a security agent. Missing one node for an agent like that means a blind spot in monitoring or a hole in your defenses. Check how many DaemonSets are already running in any cluster's system namespace:

kubectl get daemonset -n kube-system

In kind you'll see at least kindnet (the network plugin) and kube-proxy.

How it differs from Deployment

For Deployment, a person sets spec.replicas and decides how many copies to run. DaemonSet has no replicas field in its schema at all.

There's a trap hiding here. A patch that sets a replica count on a DaemonSet looks like this:

kubectl patch daemonset node-agent -p '{"spec":{"replicas":5}}'

Warning: unknown field "spec.replicas"
daemonset.apps/node-agent patched (no change)

There's no error, exit code 0. The server warned about an unknown field, dropped it, and changed nothing. A newcomer reads the word patched and assumes the replica count took effect. What matters is the Warning and the (no change) in parentheses, and you confirm the result with kubectl get daemonset.

Other ways to set a copy count fail outright. kubectl scale daemonset node-agent --replicas=5 answers Error from server (NotFound): the server could not find the requested resource: DaemonSet has no scale subresource. kubectl apply of a manifest with spec.replicas fails with a strict decoding error: unknown field "spec.replicas".

What kubectl get daemonset shows

kubectl get daemonset node-agent

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-agent   1         1         1       1            1           <none>          3s

• DESIRED: how many nodes qualify under the placement conditions.
• CURRENT: how many of them already have a Pod created.
• READY: how many Pods passed their readiness checks.
• UP-TO-DATE: how many Pods match the current template version.
• AVAILABLE: how many have been ready longer than minReadySeconds.
• NODE SELECTOR: the active nodeSelector, or <none> if there isn't one.

The number in the DESIRED column always matches the number of qualifying nodes. Check it against the output of kubectl get nodes.

nodeSelector narrows the set of nodes

nodeSelector in the Pod template keeps only nodes with the matching labels. A real-world use: a GPU-machine agent isn't needed on regular nodes.

kubectl patch daemonset node-agent \
  -p '{"spec":{"template":{"spec":{"nodeSelector":{"disk":"ssd-nvme"}}}}}'

kubectl get daemonset node-agent

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-agent   0         0         0       0            0           disk=ssd-nvme   12s

If no node has that label, DESIRED becomes zero. The controller deletes the old Pod and creates no new ones. There's no Pending and no error, because there's nowhere to create a Pod. kubectl get events will only show a SuccessfulDelete line. Node labels show up with kubectl get nodes --show-labels.

It works the other way too. Remove the label a nodeSelector is configured for from a node, and that node's DaemonSet Pod disappears by itself.

You can remove the field again with a patch that sets it to null:

kubectl patch daemonset node-agent \
  -p '{"spec":{"template":{"spec":{"nodeSelector":null}}}}'

Nodes with taints and tolerations

A taint is a mark on a node that repels Pods. A typical example: control-plane is marked with node-role.kubernetes.io/control-plane:NoSchedule, so user applications don't end up there. A monitoring agent needs to get in there, though, so you add a toleration to its template:

tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

A node with a taint the Pod doesn't tolerate isn't counted in DESIRED. So the picture is the same as with a missing label: zero, with no Pending and no errors.

In a kind cluster with one node, the control-plane taint is removed, otherwise no application could land there at all. Check it yourself:

kubectl get node -o jsonpath='{.items[0].spec.taints}'

The output is empty. That's why you don't see the difference on a cluster like this, while on a multi-node cluster a DaemonSet without a toleration quietly skips the control plane.

Two things about taints that surprise people in practice:

• The NoSchedule effect only applies to new Pods. Put a taint on a node where a DaemonSet Pod is already running, and it stays put.
• DaemonSet adds some tolerations to the Pod itself. Among them is node.kubernetes.io/unschedulable:NoSchedule, which is why kubectl cordon doesn't stop its Pod: the deleted Pod gets recreated right away on the same node. See the full list in kubectl get pod <name> -o yaml, the tolerations section.

Updates

When the Pod template changes, Pods get recreated according to spec.updateStrategy. By default that's RollingUpdate with maxUnavailable: 1 and maxSurge: 0: the controller stops one Pod, brings up a new one on the same node, waits for it to become ready, and moves on to the next one. You can raise maxUnavailable so a rollout across a hundred nodes doesn't drag on for hours.

The second strategy, OnDelete, doesn't create new Pods on its own. The new template only applies to a node after you delete the old Pod by hand.

You track a rollout with the same commands as for Deployment:

kubectl rollout status daemonset/node-agent

daemon set "node-agent" successfully rolled out

kubectl rollout history and kubectl rollout undo also work for DaemonSet.

Common mistakes

• Setting a copy count through replicas. The patch goes through with patched (no change), and DESIRED doesn't move.
• A typo in nodeSelector, or a label that never got put on the node. DESIRED 0, no Pod, no errors. Check the NODE SELECTOR column against kubectl get nodes --show-labels.
• Removing a taint from a node just to get an agent onto it. The taint disappears for every Pod at once, and applications that have no business there move in. The right move is adding a toleration to the DaemonSet template instead.
• A toleration with operator: Exists and no key. A Pod like that tolerates literally any taint, when you usually need one specific one.
• Expecting cordon to kick an agent off a node. DaemonSet tolerates unschedulable and brings the Pod right back.

What you'll need for the tasks

• kubectl get daemonset and what the DESIRED and NODE SELECTOR columns mean.
• kubectl get nodes --show-labels and kubectl describe node, the Taints section.
• Editing a DaemonSet template with kubectl edit or kubectl patch.
• The shape of a tolerations block: key, operator, value, effect.

The tasks live in the labs/ folder: each lab brings up its own kind cluster on your machine, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 3 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-daemonset     # set up the starting state
./run.sh verify k8s-daemonset    # check the current step

In short

• DaemonSet keeps one Pod on every node that qualifies; it has no replicas field.
• A patch with replicas answers Warning: unknown field and patched (no change), and kubectl scale doesn't work at all.
• DESIRED equals the number of nodes that pass nodeSelector and the taint check.
• A label that no node has zeroes out DESIRED, with no errors and no Pending.
• Without a toleration, a Pod won't land on the control plane or any other tainted node.
• Updates follow updateStrategy, one node at a time by default.