Requests and limits: CPU and memory

How requests differ from limits, where the m and Mi units come from, why CPU gets throttled but memory kills the container.

Why this matters

Dozens of Pods live on one node. Without declared resources, the scheduler places them blindly, and one hungry container can eat up all the node's memory along with its system daemons. Requests and limits solve both problems. They're the first thing you check when a Pod is stuck in Pending or restarting with OOMKilled.

requests and limits: what they're for

Each container in a Pod can declare two numbers for each resource (CPU and memory):

• requests: how much the container asks for up front. The scheduler uses it when picking a node.
• limits: the ceiling the container can't exceed while running.

containers:
  - name: app
    image: busybox:1.36.1
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 256Mi

Without requests and limits, the scheduler treats the container as "zero" and places it on a node with any free room at all. If you set only a limit, Kubernetes fills in a request equal to the limit. Without limits, a container can eat up all the node's RAM and hurt its system daemons.

Field	Who reads it	When it applies	What happens if violated
requests	kube-scheduler	once, at placement	no node fits it, Pod stays Pending
limits	kubelet and the Linux kernel	the whole time it runs	CPU gets throttled, memory gives OOMKilled

CPU units: m

CPU is measured in millicpu (m). 1000m == 1 CPU == 1 core on the node. Fractions: 500m is half a core, 100m a tenth. You can also write it as a plain number: cpu: 1 is 1000m, cpu: 0.5 is 500m.

This leads to a common typo. cpu: 250 without m means 250 cores, not a quarter of a core. The API will accept a value like this, and the Pod will simply never find a node.

Under the hood, kubelet puts the limit into a Linux CFS quota. limit 500m means "out of every 100 ms of real time, the container gets at most 50 ms of CPU." If the application wants more, it gets slowed down (throttled), but not killed.

A CPU request turns into a cgroup weight: cpu.weight in cgroup v2, cpu.shares in cgroup v1. On an idle node, a container can get more than its request if there are free cycles. Under load, the CPU is split proportionally to requests.

Memory units: Mi, Gi

Memory is measured in bytes. Suffixes:

• Ki: 1024 bytes
• Mi: 1024 × 1024 (~1 MB)
• Gi: 1024 × 1024 × 1024 (~1 GB)

Without the i, these are powers of 10 (1M = 1000000, 1G = 1000000000). Manifests almost always use the i form: 128Mi, 1Gi. For memory, requests is just a number for the scheduler, while limits is a hard boundary. Exceed the memory limit, and the kernel kills the process, giving the container the reason OOMKilled. There's no throttling here like there is for CPU.

How the scheduler picks a node

The scheduler looks at the sum of requests from every Pod already assigned to a node and compares it with the node's Allocatable. If a Pod asks for more than what's left, the scheduler skips that node. The node's actual load doesn't matter here: only declared requests count.

A Pod in status Pending with an event Insufficient cpu or Insufficient memory means "no node can fit my request." In kubectl describe pod, it looks like this:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  5s    default-scheduler  0/1 nodes are available: 1 Insufficient cpu. ...

There are two fixes: lower the requests or add nodes. In a single-node cluster, the second option doesn't exist, so you're left cutting requests.

CPU throttling and memory OOMKill

These are two different stories, and it's easy to mix them up:

• CPU went over its limit: the Linux CFS slows the process down. The program runs slower, but it keeps running. In metrics you see this as container_cpu_cfs_throttled_seconds_total > 0, and inside the container as a growing nr_throttled in the file /sys/fs/cgroup/cpu.stat.
• Memory went over its limit: the kernel's OOM killer kills the process. The container's status gets Last State: Terminated, Reason: OOMKilled, and restartPolicy decides whether to restart it. For a Deployment, the container restarts again and again, and the Pod ends up in CrashLoopBackOff.

CPU throttling often gets missed: the application complains about slow responses, while the developer is sure the limit has "plenty of room." Watch the throttle metrics, not just usage.

OOMKilled is also easy to miss: kubectl logs for a container like this usually explains nothing: the kernel kills the process without a message in its output. The reason shows up in kubectl describe pod, in the Last State field.

Going deeper: QoS classes

The core of the topic is already behind you. QoS classes are built on top of requests and limits, and you can read this section later.

Based on the combination of requests and limits, Kubernetes assigns a Pod a QoS class. It affects who gets evicted first when a node runs short on memory:

• Guaranteed: every container in the Pod has requests and limits set for both CPU and memory, and requests == limits. Evicted last.
• Burstable: doesn't qualify for Guaranteed, but at least one container has a request or limit for CPU or memory. Medium priority.
• BestEffort: no container has requests or limits at all. First in line for eviction when the node is under pressure.

The class is visible with kubectl get pod <name> -o jsonpath='{.status.qosClass}'. For a critical production app, you usually go with Guaranteed. For batch jobs, Burstable is often enough.

What commonly breaks

• Pod stuck in Pending with Insufficient cpu. Check the units (250 instead of 250m), then lower the request or add a node.
• Pod repeatedly OOMKilled. Raise the memory limit to match the real usage peak with some room, or find the leak. Removing the limit entirely is a bad idea: then a leak hurts the whole node.
• Memory limit set right at normal usage. Any spike kills the container. Set the limit by the peak, not the average.
• Changed the limits directly on the Pod instead of the Deployment. The next Pod from the Deployment's template will come back with the old numbers.

Related commands

# What's declared in the spec
kubectl get pod app -o jsonpath='{.spec.containers[0].resources}'

# Why the Pod isn't starting or keeps restarting
kubectl describe pod app

# What's actually being used (needs metrics-server)
kubectl top pod app

# How much of the node's resources are already claimed by requests
kubectl describe node <name> | grep -A8 'Allocated resources'

# Field schema
kubectl explain pod.spec.containers.resources
kubectl explain pod.spec.containers.resources.requests

What you'll need for the tasks

• kubectl get pods and kubectl describe pod <name>: status, FailedScheduling events, the Last State field with the termination reason.
• kubectl describe node and the Allocatable line: how much CPU and memory the node actually has.
• Units: m for CPU, Mi for memory, and what a number with no suffix means.
• Editing resources in a Deployment: kubectl edit deployment <name> or kubectl set resources deployment <name> ..., after which the Deployment recreates the Pod.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your computer, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-resource-requirements     # set up the starting state
./run.sh verify k8s-resource-requirements    # check the current step

In short

• requests is a number for the scheduler, limits is a ceiling while it runs.
• CPU is counted in millicpu: 1000m is one core, 250 without m is 250 cores. Memory in Mi/Gi, without a suffix the number reads as bytes.
• The scheduler compares the sum of requests on the node with Allocatable. Doesn't fit, Pod goes Pending with Insufficient cpu/memory.
• Exceeding the CPU limit gets you throttling, exceeding the memory limit gets you OOMKilled.
• The combination of requests and limits determines the QoS class, and that decides the eviction order when memory runs short.