PodDisruptionBudget: protection from drain

How controlled eviction differs from an emergency one, and how minAvailable and maxUnavailable hold back kubectl drain.

Controlled and uncontrolled evictions

A Pod is the smallest unit that Kubernetes runs. Pods can disappear for two different reasons.

Uncontrolled eviction (involuntary). A node goes offline, the OOM killer kills a process, a node loses contact with the cluster. Here Kubernetes guarantees nothing.

Controlled eviction (voluntary). Someone deliberately removes a Pod. For example, kubectl drain puts a node into maintenance and evicts every Pod on it. These operations go through the API eviction endpoint: a request to Kubernetes to delete a Pod, which first checks against the budget.

Not every deliberate deletion goes through the API eviction endpoint. kubectl delete pod and deleting a Deployment don't ask the budget. A Deployment's rolling update isn't limited by it either: maxUnavailable and maxSurge on the Deployment itself set the rollout pace. The budget counts such Pods as unavailable, but doesn't stop them from being deleted.

What PodDisruptionBudget does

PodDisruptionBudget (PDB) is an object that sets a minimum number of live Pods during controlled evictions. It tells Kubernetes: evict Pods so that the number in the Ready state never drops below the given count.

Ready is the state of a Pod in which it's ready to accept traffic. If an eviction would violate the budget, the API refuses (code 429 Too Many Requests), and kubectl drain waits, retrying the request.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

This PDB requires at least 2 Pods with the label app: api to be in Ready. If there are three, one can be evicted. If there are two, the eviction is blocked.

minAvailable: 2
  Ready ●●●  →  can evict 1
  Ready ●●   →  drain blocked

selector picks Pods by their labels. It has to match the labels the Deployment sets. If the selector doesn't match a single Pod, the PDB exists but protects nothing. What that looks like in the output is covered below.

minAvailable and maxUnavailable

A budget is set through one of two fields.

spec:
  minAvailable: 2     # minimum 2 Pods in Ready

spec:
  maxUnavailable: 1   # maximum 1 Pod not in Ready at a time

The value is set as a number or a percentage, for example "50%". A percentage is counted against the Pod count and rounded up: minAvailable: "50%" with three replicas requires 2 Pods. A single PDB uses either minAvailable or maxUnavailable, never both at once: the API won't accept an object like that.

If minAvailable equals the replica count, not a single Pod can be evicted. The service is "protected", but node maintenance becomes impossible. minAvailable: "100%" and maxUnavailable: 0 give the same result. For three replicas, set minAvailable: 2 or maxUnavailable: 1, and then one of the three Pods can be evicted.

maxUnavailable is more convenient when the replica count changes (say, driven by the HPA): a one-Pod margin holds regardless of how many replicas there are.

A PDB can be changed on the fly, kubectl edit pdb <name>. There's no need to recreate it.

How to read the state

kubectl get pdb api-pdb

NAME      MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api-pdb   2               N/A               1                     30s

ALLOWED DISRUPTIONS shows how many Pods can be evicted right now. With three Ready and minAvailable: 2, that number is 1. If it's 0, eviction is blocked.

kubectl describe pdb api-pdb

Name:           api-pdb
Namespace:      default
Min available:  2
Selector:       app=api
Status:
    Allowed disruptions:  1
    Current:              3
    Desired:              2
    Total:                3
Events:                   <none>

Current shows how many Pods under the budget are currently in Ready. Desired shows how many the budget requires. Total shows how many Pods the budget found by its selector in total.

Total is the first thing worth checking. ALLOWED DISRUPTIONS 0 happens for two reasons: a strict budget, or a budget that found no Pods at all. In the second case, Total is 0, and drain will evict all of the service's Pods with no limit.

How PDB behaves during drain

kubectl drain needs permissions on nodes, so it's usually an admin who runs it. drain first marks the node as closed to new Pods (cordon), then evicts the existing ones. drain won't evict DaemonSet Pods and refuses to run at all without the --ignore-daemonsets flag.

A PDB will make drain get stuck on the last allowed Pod. Here's what that looks like:

kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data

node/node-2 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kindnet-..., kube-system/kube-proxy-...
evicting pod default/api-7d4f-aaa
error when evicting pods/"api-7d4f-aaa" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod default/api-7d4f-aaa
...

By default, drain waits forever. The --timeout=60s flag makes it give up and exit with an error. The node stays closed off regardless.

After maintenance, the node is returned to service with kubectl uncordon <node>, otherwise new Pods won't be able to land on it. The evicted Pods don't move back on their own: they're already running on other nodes.

What often breaks

• minAvailable equals the replica count (or "100%"). drain gets stuck on the very first Pod.
• The budget's selector doesn't match the Pod's labels. The budget exists, Total is 0, there's no protection.
• A budget on a single-replica Deployment with minAvailable: 1. You can't drain the node with that Pod until you add more replicas.
• People expect a PDB to save them from a node going down. It won't: that's an uncontrolled eviction.
• A stuck drain gets interrupted with Ctrl+C and the node is forgotten. The cordon already happened, the node is closed, until you run uncordon.

What you'll need for the tasks

• kubectl get pdb: the ALLOWED DISRUPTIONS column.
• kubectl describe pdb <name>: the Selector, Current, Desired, Total lines.
• kubectl get pods --show-labels: a Pod's labels, to compare against the selector.
• kubectl edit pdb <name> or kubectl patch pdb: change a budget on the fly.
• kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=60s and kubectl uncordon <node>.
• kubectl get nodes: SchedulingDisabled on a closed node.
• For the mini-project: requests and limits, kubectl label nodes, kubectl taint nodes, tolerations and nodeAffinity from earlier lessons in the module.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your machine, instructions in its README.md. The last lab, the mini-project, pulls the whole module together into one task.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-pdb     # set up the starting state
./run.sh verify k8s-pdb    # check the current step

In short

• PDB limits only controlled evictions through the API eviction endpoint. It won't save you from a node going down, kubectl delete pod, or a rolling update.
• A budget is set through either minAvailable or maxUnavailable, as a number or a percentage. You can't use both fields at once.
• selector has to match the Pod's labels, otherwise the PDB protects nothing. Check Total in kubectl describe pdb.
• ALLOWED DISRUPTIONS in kubectl get pdb shows how many Pods can be removed right now. Zero means drain will get stuck.
• minAvailable equal to the replica count makes node maintenance impossible.