Node Affinity: required and preferred

How to tell the scheduler "only on nodes like this" and "better on nodes like this, but others are fine too", and where OR hides in the YAML.

Why this matters

nodeSelector only knows strict equality: a key and a value. When both an SSD node and an NVMe node work, it can't express that. "Zone A is better, but anything will do" can't be expressed either: nodeSelector either admits the Pod to a node or it doesn't. nodeAffinity exists for rules like these.

What nodeAffinity is

nodeAffinity solves the same problems as nodeSelector, but does more: operators beyond equality, soft rules (preferences) and OR between groups of conditions. The field lives in spec.affinity.nodeAffinity on a Pod, and in a Deployment it's in the Pod template: spec.template.spec.affinity.nodeAffinity.

nodeSelector can be written through affinity: it's a required rule with one In expression and one value. If a Pod has both fields set, the node has to match both nodeSelector and affinity.

Required and preferred

Two modes:

• requiredDuringSchedulingIgnoredDuringExecution. A hard requirement. Without a matching node, the Pod stays Pending. For critical conditions: a GPU app without a GPU node won't start at all.
• preferredDuringSchedulingIgnoredDuringExecution. A preference. Each rule has a weight from 1 to 100. The scheduler adds up the weights of the rules that matched for each node and adds the sum to the node's other scores. If no node matches the preferences, the Pod still lands wherever there's room. Good for optimizations: "better close to the database", "better in zone A, but B works too".

IgnoredDuringExecution means: if the node's labels change after the Pod starts, the Pod isn't touched. The rules are checked only at placement time.

The variant requiredDuringSchedulingRequiredDuringExecution, which would evict the Pod when labels change, is documented as planned and isn't implemented in Kubernetes.

Minimal YAML

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: disk
                operator: In
                values: [ssd, nvme]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 50
          preference:
            matchExpressions:
              - key: zone
                operator: In
                values: [a]

Required and preferred can be set together. Required filters the nodes (only SSD or NVMe), preferred raises the score of nodes with zone=a.

Note the shape. In required, the rules sit inside nodeSelectorTerms, which is a list. In preferred, there is a list of rules, each with its own weight and one preference.

nodeSelectorTerms: where it's AND, and where it's OR

nodeSelectorTerms is an array. Between different terms it's OR: a node only needs to match any one of them. Inside one term, expressions from matchExpressions are combined with AND: a node needs to match all of them at once.

So a term is an AND block, and an array of terms is an OR block. "Either a node with GPU, or a node with the label fallback=cpu" is written as two terms:

nodeSelectorTerms:
  - matchExpressions:
      - key: gpu
        operator: Exists
  - matchExpressions:
      - key: fallback
        operator: In
        values: [cpu]

If you need OR on values of one key, a single expression with a list is enough: operator: In and values: [ssd, nvme].

matchExpressions operators

• In. The node's label value is in the values list.
• NotIn. It isn't. For example, "don't place on spot instances".
• Exists. The node has this key, the value isn't checked. You don't write values.
• DoesNotExist. The node doesn't have this key.
• Gt, Lt. Numeric comparison for labels like cpu-cores=16. values has exactly one value, and it's an integer in quotes: ["8"].

NotIn and DoesNotExist give you "anti-affinity to nodes" without taints. That's handy when you're not a cluster admin, can't set a taint on nodes, but want to stay away from some group of them. NotIn also matches a node without the key at all: there's no label, so it isn't equal to the banned value.

How this looks in the cluster

The effect of preferred rules only shows up where the scheduler actually has a choice. In a single-node cluster, scoring changes nothing: the Pod lands on the only node there is, even if no preference matched. Required works there too, though: if a label doesn't match, the Pod stays Pending.

The reason is visible in kubectl describe pod, the Events section, the same text as for nodeSelector:

Warning  FailedScheduling  ...  default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. ...

If a Pod is stuck in Pending and you add the missing label to a node, the scheduler places it on its next attempt. But if the Pod is already running, changing labels won't move it. To make the rules apply again, you need a new Pod: for a Deployment that's kubectl rollout restart deployment/<name>.

Common mistakes

• Required is too strict. No node matches, the Pod sits in Pending, and nobody notices until there's an incident. If the condition isn't critical, use preferred with a high weight.
• Only preferred for a critical condition. The scheduler puts the Pod anywhere, and then monitoring complains the service isn't on the right node.
• Two expressions under one - matchExpressions: where you need OR. They get combined with AND (see the section on nodeSelectorTerms).
• The node's labels changed, but the Pod stayed where it was. That's by design: IgnoredDuringExecution. The Pod needs a restart.
• Gt with an unquoted value (values: [8]): values is a list of strings, the API will reject the manifest.

Related commands

kubectl get nodes -L disk -L zone
kubectl get nodes --show-labels
kubectl get pod <name> -o jsonpath='{.spec.affinity}'
kubectl get deployment <name> -o yaml        # the spec.template.spec.affinity section
kubectl describe pod <name>                  # the Events section, if Pending
kubectl rollout restart deployment/<name>    # recreate the Pod under the new rules
kubectl explain pod.spec.affinity.nodeAffinity

What you'll need for the tasks

• kubectl get nodes -L <key>: which labels are actually on the node.
• kubectl get deployment <name> -o yaml: where affinity lives and how expressions are split across terms.
• kubectl describe pod <name>: the FailedScheduling event.
• The rule "AND inside a term, OR between terms".
• The In and NotIn operators, the shape of a preferred rule with weight.
• kubectl label nodes <node> key=value and removing a label with key-, kubectl rollout restart deployment/<name>.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your machine, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-node-affinity     # set up the starting state
./run.sh verify k8s-node-affinity    # check the current step

In short

• nodeAffinity extends nodeSelector: operators, preferences and OR.
• Required filters nodes, preferred only adds points to the score.
• nodeSelectorTerms combines terms with OR, matchExpressions inside a term with AND.
• Operators: In, NotIn, Exists, DoesNotExist, Gt, Lt.
• IgnoredDuringExecution: changing a node's labels doesn't move a Pod that's already running.