Taints and nodeAffinity: pass and address

Why a toleration without nodeAffinity doesn't keep a Pod on the dedicated node, and nodeAffinity without a toleration leaves it Pending.

Why compare taints and affinity

At first glance they do the same thing: restrict where a Pod runs. In practice they work from two different sides, which is exactly why they're often used together.

Short rule:

Taint = pass: only a Pod with a matching toleration is let onto the node. Affinity = address: the Pod itself looks for a node with the right label.

To make sure a Pod lands on a dedicated node and doesn't drift elsewhere, you need both mechanisms. Either one alone breaks.

Mechanism	Who sets it	What it gives	What it doesn't give
taint + toleration	admin on the node, developer in the Pod	lets the Pod onto a tagged node	doesn't pull the Pod toward it
label + nodeAffinity	admin on the node, developer in the Pod	aims the Pod at a node	doesn't remove the taint

A taint and a label on a node are independent of each other. Setting a taint doesn't create a label, and the other way around.

Toleration only: the Pod goes wherever

A toleration lets a Pod ignore a taint. It doesn't tell the scheduler "put me here", only "if you put me here, I'm fine with it".

In a cluster with several nodes, a Pod with a toleration but no affinity can land on any node that passes the checks. Nodes without taints suit it just as well as the dedicated one. The dedicated node stays empty, even though resources were set aside for it.

Affinity only: the Pod has the address but isn't let in

Affinity does the opposite: it requires a node with the right label, but doesn't remove the taint from it. The scheduler checks every node against every filter. A node without the label drops out on affinity. A node with the label drops out on a taint the toleration doesn't cover. No nodes are left, the Pod sits Pending.

This is a classic trap: the Pod's affinity is correct, the node's label is correct, and the Pod is still Pending. The cause is a taint the toleration doesn't cover: it's missing entirely, or one of its fields doesn't match the taint.

The full set

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: critical
      effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: dedicated
                operator: In
                values: [critical]

The Pod tolerates the taint dedicated=critical:NoSchedule and itself looks for a node with the label dedicated=critical. Both requirements point to the same node, and the Pod passes both filters. In a Deployment, this block lives in spec.template.spec, next to containers.

A nodeSelector (see the nodeSelector lesson) can serve as the address instead of required nodeAffinity: for a single exact label match, it's enough.

Scenario: a GPU pool for ML only

A typical setup:

1. The admin sets a taint: kubectl taint nodes gpu-pool dedicated=gpu:NoSchedule. Random Pods with no toleration won't land here.
2. The admin sets a label: kubectl label nodes gpu-pool dedicated=gpu. It can be used as an affinity target.
3. ML apps add both a toleration for dedicated=gpu:NoSchedule and nodeAffinity on dedicated In [gpu] to their Pods.

The GPU nodes are protected from outsiders, and the ML Pods land on the right hardware.

How the scheduler handles them

kube-scheduler picks a node in two stages: filtering and scoring.

Filtering drops nodes the Pod can't land on:

• a taint with no matching toleration;
• no label for required nodeAffinity or nodeSelector;
• not enough resources for the requests.

Scoring gives points to the nodes that remain:

• preferred nodeAffinity adds points to nodes with the right labels;
• built-in rules count too: spreading load evenly across nodes, whether a node already has the image, and others.

If filtering drops every node, the Pod stays Pending. If several nodes pass, the one with the highest score is picked.

In a single-node cluster you only see half the picture. The filter on taints and required affinity works, but the "toleration present, affinity missing" mismatch never shows up: the Pod has nowhere else to go anyway. The difference between the two mechanisms only shows when there are at least two nodes. That's why this lesson's labs bring up a two-node cluster.

Common mistakes

• Set only a toleration, forgot affinity. The Pod drifted to an ordinary node. The dedicated one sits idle, the budget goes to waste.
• The label and the taint use different keys, say taint dedicated=critical and label tier=critical. It can work, but it's easy to write the toleration and the affinity against the same key, and have one of them match nothing. One key and one value is simpler to keep track of.
• A typo or a different value in the toleration. The API won't notice, the taint stays uncovered, the Pod is Pending.
• Mixed up the effect. The toleration is for NoSchedule, but the node's taint is NoExecute. The toleration doesn't cover the taint, the Pod won't land.
• "Fixed" the Pending by removing affinity. The Pod started, but on the shared node.

Related commands

kubectl taint nodes <name> <key>=<value>:<effect>
kubectl label nodes <name> <key>=<value>
kubectl get nodes -L <label-key>
kubectl describe node <name> | grep -A3 Taints
kubectl describe pod <name> | grep -A10 Events
kubectl explain pod.spec.tolerations
kubectl explain pod.spec.affinity.nodeAffinity

What you'll need for the tasks

• kubectl get pods -o wide: the NODE column shows where the Pod landed.
• kubectl describe pod <name>: the FailedScheduling event with a reason for each node.
• kubectl describe node <node> | grep -A3 Taints and kubectl get nodes --show-labels: what's set on a node.
• kubectl label nodes <node> <key>=<value>: set the address.
• Where tolerations and affinity live in a Deployment: spec.template.spec, next to containers.
• Rule: toleration lets in, affinity directs; a dedicated node needs both.

The tasks are in the labs/ folder: each lab brings up its own two-node kind cluster on your machine, instructions in its README.md.

In short

• A toleration lets a Pod land on a node with a taint, but doesn't pull it there.
• nodeAffinity aims a Pod at a node with the right label, but doesn't remove the taint.
• A dedicated node needs both: taint + toleration and label + affinity, easier to manage on one key.
• The scheduler filters nodes first (taints, required affinity, resources), then scores what's left (preferred affinity and built-in rules).
• Pending with untolerated taint in the event means the toleration is missing or doesn't match the taint. Affinity has nothing to do with it.