Taints and tolerations: who gets onto a node

How a node refuses Pods it isn't expecting on its own, how a Pod gets permission to land on it, and how a taint evicts Pods that are already running.

Why this matters

nodeSelector and nodeAffinity solve the problem from the Pod's side: "I want this kind of node." There's also the reverse problem. Some nodes have GPUs, some are for the database, and one gets taken down for maintenance tomorrow. You can't let just anyone land on them, and adding a selector to every unrelated Deployment isn't an option. That's what a taint on a node is for.

What taints are

A taint is a mark on a node in the format key=value:effect. When choosing a node, the scheduler reads its taints and checks them against the Pod's tolerations. If even one taint isn't covered by a toleration, the node is dropped from the candidates, and the Pod won't land on it.

This is the inverse of nodeSelector. There, the Pod picks a node by labels; here, the node filters Pods. A taint means "don't let anyone in except those who explicitly said they're ready." That's how dedicated nodes work: a GPU pool, nodes for system Pods, isolated workloads.

An admin sets taints with the kubectl taint command. A developer writes tolerations in the Pod spec. The link runs both ways: the admin says "only workload=stateful goes here," and the developer adds the matching toleration to the workloads that need it.

Three effects

• NoSchedule. New Pods without a toleration won't land on the node. Pods already running stay put: the taint arrived after them.
• PreferNoSchedule. A soft ban. The scheduler tries to avoid the node, but if nothing else fits, it will place the Pod there anyway.
• NoExecute. New Pods without a toleration won't land, and running ones get evicted. The decision isn't made by the kubelet but by the taint-eviction-controller inside kube-controller-manager. A Pod with no matching toleration is deleted right away. A Pod with a toleration that sets tolerationSeconds stays on the node for that many seconds, then is also evicted.

NoExecute is what evacuation from failed nodes runs on. When a node stops responding, the control plane puts the taint node.kubernetes.io/not-ready or node.kubernetes.io/unreachable on it with effect NoExecute. Every Pod automatically gets tolerations for these two taints at creation time, with tolerationSeconds: 300. That's why Pods leave a failed node after about five minutes, not instantly.

Minimal toleration YAML

spec:
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "stateful"
      effect: "NoSchedule"

Field by field:

• key: the key of the node's taint the Pod tolerates.
• operator: Equal (the value must match) or Exists (any value fits, and the toleration doesn't set a value at all). Equal by default.
• value: the right-hand side of the taint. Written only with Equal.
• effect: which effect the Pod tolerates. If this field is empty, the toleration matches the taint with this key and value under any effect.
• tolerationSeconds: only for NoExecute. How many seconds the Pod stays on the node after the taint appears before it's evicted.

A toleration covers a taint when the key and effect match, and the value matches too with Equal (with Exists, the value isn't compared).

Operator: Equal and Exists

Equal requires the value to match. Exists catches any taint with the right key. If you set a value with Exists, the API rejects the manifest. The "tolerate everything" template:

tolerations:
  - operator: "Exists"

No key, no effect. A toleration like this matches any taint. It's used on system DaemonSets (a controller that keeps one Pod per node): the network plugin, kube-proxy, node-exporter, a log collector. They need to run on every node, including the control plane, which usually carries the taint node-role.kubernetes.io/control-plane:NoSchedule.

The admin sets and removes taints

kubectl taint nodes node-1 workload=stateful:NoSchedule
kubectl taint nodes node-1 workload-
kubectl describe node node-1 | grep -A2 Taints

The second command removes every taint with the key workload from the node, whatever its effect. To remove one specific taint: kubectl taint nodes node-1 workload=stateful:NoSchedule-. You can change the value of an existing taint only with the --overwrite flag.

In a single-node cluster, a taint is especially easy to see. Mark the node NoSchedule, and any new Pod without a toleration sits in Pending, because the scheduler has no other candidates. In the Pod's events it looks like this:

Warning  FailedScheduling  ...  0/1 nodes are available: 1 node(s) had untolerated taint(s). ...

A single-node kind cluster doesn't put a taint on the control plane, otherwise no ordinary Pod would land there at all. You can check this with kubectl describe node: the Taints line will show <none>.

Common mistakes

• You wrote a toleration but forgot nodeAffinity. A toleration only allows landing on the dedicated node, it doesn't pull the Pod there. Without affinity, the Pod will happily go to any other node. The lesson on taints and nodeAffinity covers how to combine the two.
• A typo in the key or value. The API won't notice: to it, this is just an ordinary string. The scheduler simply won't match the toleration to the taint, and the Pod stays in Pending.
• Mixing up Equal and Exists. With Exists you don't write a value, with Equal you must.
• You set a taint with NoSchedule and expect running Pods to leave. They won't: only NoExecute evicts.
• You added a "tolerate everything" toleration without a key to the Pod spec to quickly fix a Pending. The Pod starts, but now it will also land on the GPU node and the one under maintenance.

Related commands

kubectl taint nodes <name> <key>=<value>:<effect>
kubectl taint nodes <name> <key>-
kubectl describe node <name> | grep -A3 Taints
kubectl get node <name> -o jsonpath='{.spec.taints}'
kubectl get pod <name> -o jsonpath='{.spec.tolerations}'
kubectl explain pod.spec.tolerations
kubectl explain node.spec.taints

What you'll need for the tasks

• kubectl describe node <node> | grep -A3 Taints: which taints are on the node.
• kubectl describe pod <name>: the FailedScheduling event and the mention of untolerated taint.
• kubectl get deployment <name> -o yaml: tolerations live in spec.template.spec.tolerations, next to containers.
• The matching rule: a toleration's key, effect, and (with Equal) value must match the taint.
• kubectl taint nodes and the difference between NoSchedule and NoExecute for Pods that are already running.

Tasks are in the labs/ folder: each lab brings up its own kind cluster on your computer, with instructions in its README.md.

In short

• A taint is a mark on a node, key=value:effect; it keeps out Pods without a matching toleration.
• Three effects: NoSchedule (block new ones), PreferNoSchedule (block if possible), NoExecute (block new ones and evict running ones).
• A toleration in the Pod spec is described with key, operator, value, effect; tolerationSeconds only makes sense with NoExecute.
• operator: Exists with no key or effect tolerates any taint; system DaemonSets rely on this.
• A toleration only allows landing on a node. It can't pull a Pod toward it.