Pod Pending because of CPU

What FailedScheduling means, how requests differ from limits, and why a node with 4 cores won't give a Pod 4 cores.

Why this matters

A Pod is created but doesn't start. A minute, five minutes, an hour, the status doesn't change.

NAME    READY   STATUS    RESTARTS   AGE
heavy   0/1     Pending   0          2m

Pending means: the object exists in the API, but the container hasn't started yet. Most often the Pod is stuck before it even starts, at the node-selection stage. That's the job of the scheduler, a control plane component that picks a node for every new Pod. If there's no node that fits, the Pod will wait indefinitely, with no errors and no restarts. You need to be able to recognize this situation in a minute.

Why a Pod gets stuck in Pending

The Pending phase lasts from the Pod's creation until all its containers start. Several different causes fall into it, and the STATUS column in kubectl get pods tells them apart:

1. The scheduler didn't find a node. Not enough resources, taints get in the way, affinity or nodeSelector didn't match, a PVC isn't bound. STATUS: Pending.
2. The image is still downloading. A node is already chosen, and kubelet is pulling a large image. STATUS: ContainerCreating.
3. An init container is running. The main containers are waiting for it to finish. STATUS: Init:0/1.

The first case is the most common, and it's also the least obvious at first glance: nothing is wrong with the Pod, there's just nowhere to put it. The rest of this lesson is about that case.

What the diagnostics show

kubectl describe pod heavy

The main section in the output is Events. When there's a problem choosing a node, you'll see a FailedScheduling event there:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  12s   default-scheduler  0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Read the message left to right:

• 0/1 nodes are available means none of the 1 node worked.
• 1 Insufficient cpu names the reason and the number of nodes that were ruled out for it. If there are several nodes and several reasons, they're listed with commas.
• The tail preemption: ... says the scheduler tried to free up space by evicting a lower-priority Pod, and found no one to evict.

Common reasons:

The exact same Insufficient cpu happens in two different situations. The Pod either waits for space to free up on the node, or it will never fit, because it asks for more than the biggest node has. You can only tell them apart by the node's numbers, more on that below.

Requests vs. limits

requests is how much the Pod is guaranteed to get. When picking a node, the scheduler looks only at requests. The sum of requests.cpu of all Pods on a node can't exceed allocatable (more on that below).

limits sets a ceiling the container can't rise above while running. On their own, limits don't affect node selection.

There's a catch. If a container has a limit set but no request, Kubernetes copies the limit into the request. A manifest with just limits.cpu: 8 is read by the scheduler as a request for 8 cores. You can check what actually happened on the created Pod:

kubectl get pod heavy -o jsonpath='{.spec.containers[0].resources}'

{"limits":{"cpu":"8"},"requests":{"cpu":"8"}}

If a Pod requests requests.cpu: 64, the scheduler looks for a node with at least 64 CPU free. In a cluster of 4-core nodes, no such node exists, and the Pod stays in Pending forever.

CPU and memory units

CPU:

• 1 or 1000m means 1 CPU (one core, or one vCPU in the cloud).
• 100m means 0.1 CPU, a tenth of a core.
• 2.5 means 2.5 CPU.

m stands for millicpu, a thousandth of a CPU. It has nothing to do with megahertz.

Memory:

• Mi is a mebibyte (1024 KiB), Gi is a gibibyte (1024 MiB).
• M and G are megabyte and gigabyte, based on 1000. You can use them, but they're harder to do mental math with.

Without a suffix, CPU is whole cores and memory is bytes. So cpu: 64 reads as "give me 64 cores", while cpu: 64m reads as sixty-four thousandths of a core. One letter changes the request a thousand times over.

Capacity vs. allocatable

kubectl describe node shows two different sets of numbers:

Capacity:
  cpu:                4
  memory:             8Gi
Allocatable:
  cpu:                3800m
  memory:             7Gi

Capacity shows how much the node physically has. Allocatable is usually smaller: the administrator reserves part of the resources for kubelet, the container runtime, and system daemons. The scheduler counts by allocatable. On local clusters the reserve is often not configured, and the numbers match.

Further down in the same output, you can see what's already taken:

Allocated resources:
  Resource           Requests     Limits
  --------           --------     ------
  cpu                500m (13%)   1 (26%)
  memory             256Mi (3%)   512Mi (7%)

These are the sums of requests and limits of all Pods on the node, not the real load. The free space the scheduler sees is calculated as Allocatable - Requests from Allocated resources. If a new Pod asks for more, the scheduler replies Insufficient cpu.

allocatable  3800m  ████████████████████
in use        500m  ███
free         3300m  █████████████████
Pod asks    64000m  doesn't fit

Two conclusions follow. A Pod that asks for more than allocatable won't fit even on an empty node. A Pod that asks for less than allocatable, but more than what's free, will run as soon as space opens up on the node.

How to fix it

There are three ways.

First: lower the new Pod's requests to its real consumption. Rough but workable guides: 100m for a light service, 500m for a medium one, 2 for a heavy one. Exact numbers come from metrics, for example kubectl top pod on a running copy (needs metrics-server).

Second: free up space. Often a node isn't taken up by real load, but by neighbors' inflated requests. Allocated resources and the list of Pods on the node show who reserved how much.

Third: add a node. In the cloud this is done by the cluster autoscaler, which reacts to exactly this kind of Pending Pod.

If the Pod was created by a Deployment, edit the Deployment's template, and the Pods recreate themselves:

kubectl set resources deployment web --requests=cpu=200m

A standalone Pod is easier to delete and recreate with a fixed manifest. A plain kubectl apply with new resources on an existing Pod is rejected by the API. Starting with Kubernetes 1.35, a running Pod's CPU and memory can be changed in place through the resize subresource (kubectl patch pod ... --subresource resize), but that's a separate topic.

Useful commands when investigating:

# The reason is at the end of describe
kubectl describe pod heavy | tail -10

# How much the node has and how much is taken
kubectl describe node | grep -A5 "Allocatable:"
kubectl describe node | grep -A5 "Allocated resources"

# How much the Pod requests
kubectl get pod heavy -o jsonpath='{.spec.containers[0].resources}'

Common mistakes

• Setting requests "just in case, with a margin". Inflated requests eat up space on nodes, and the cluster hits its limit while only 20 percent loaded.
• Assuming that since only limits are set, the scheduler doesn't see them. Without a request, the limit becomes the request too.
• Writing cpu: 500 instead of cpu: 500m.
• Waiting for a Pod with a request bigger than allocatable to eventually run on its own.
• Editing resources on a Pod created by a Deployment. The Deployment will restore the old value on the next Pod; edit the template instead.

To catch these mistakes earlier, namespaces get a LimitRange with default and maximum values. Then a Pod without requests gets reasonable numbers, and a Pod with cpu: 64 is rejected by the API at creation, with a clear error instead of a silent Pending.

In short

• Pending in STATUS most often means the scheduler hasn't chosen a node. The reason is in Events as FailedScheduling.
• The format 0/X nodes are available: <reason> reads literally.
• The scheduler counts only requests. A limit affects placement only when no request is set: then it's copied from the limit.
• Free space on a node is Allocatable - Requests, not what the real load shows.
• A standalone Pod gets recreated; for a Deployment, edit the template.

What you'll need for the tasks

• kubectl config current-context, to make sure you're working in the lab's cluster.
• kubectl get pods and kubectl describe pod <name>, the Events section.
• kubectl describe node, the Allocatable and Allocated resources blocks.
• kubectl get pod <name> -o jsonpath='{.spec.containers[0].resources}'.
• kubectl set resources deployment <name> --requests=cpu=<value>.