cgroups: resource limits

Namespaces decide what a process can see. A different kernel mechanism, cgroups, decides how many resources it can take. This lesson covers where the limits live, how Docker sets them, and what happens when a process hits one.

Namespaces are about visibility, cgroups are about resources

In the previous lesson, namespaces defined what a process sees: its own process list, its own network, its own filesystem. But seeing only a slice of the system doesn't mean using its resources modestly. A process with a cut-down view can still grab every CPU core and all the host's memory.

cgroups (control groups) is a kernel mechanism that limits how many resources a group of processes can consume: CPU time, memory, disk I/O, number of processes. Namespaces answer "what do I see", cgroups answer "how much am I allowed to take".

The group hierarchy

Processes are grouped into a tree. Each group is a directory in a special filesystem mounted at /sys/fs/cgroup. Files inside the directory hold limits and current counters. Each resource type has its own controller: memory, cpu, io, pids.

The current version is cgroup v2, a single hierarchy for all controllers. The older cgroup v1, with a separate directory per controller, is legacy now: current distributions (Ubuntu 22.04+, Debian 11+, Fedora 31+, RHEL 9) boot with v2 by default. Check which one you have by the filesystem type:

$ stat -fc %T /sys/fs/cgroup
cgroup2fs                  # v2; tmpfs means the old v1

The file cgroup.procs holds the PIDs of the group's processes, and files like memory.max, cpu.max, and pids.max set the limits. The easiest place to look is from inside a container: there, /sys/fs/cgroup shows the container's own group.

$ docker run --rm busybox:1.36.1 ls /sys/fs/cgroup
cgroup.controllers  cpu.max  memory.current  memory.max  pids.max  ...
$ docker run --rm busybox:1.36.1 cat /sys/fs/cgroup/memory.max
max                 # max means "no limit"
$ docker run --rm busybox:1.36.1 cat /sys/fs/cgroup/cpu.max
max 100000          # first number is the quota, second is the period in microseconds

At the root of the hierarchy on the host, these files don't exist. Limits apply only to child groups; the root is always unlimited. That's why cat /sys/fs/cgroup/memory.max on the host answers No such file or directory.

cpu.max reads like this: per period, the group can spend at most the quota of CPU time. 50000 100000 means 50 ms out of every 100 ms, so half of one core. 200000 100000 gives two cores.

How this connects to containers

Resource-limiting flags on docker run create a cgroup under the hood and write numbers into it. --memory 512m sets memory.max, --cpus 1 configures cpu.max, and --pids-limit 100 writes pids.max:

docker run -d --name web --memory 512m --cpus 1 --pids-limit 100 nginx:1.27.3-alpine

From inside the container you see exactly these numbers:

$ docker exec web cat /sys/fs/cgroup/memory.max
536870912           # 512 * 1024 * 1024 bytes
$ docker exec web cat /sys/fs/cgroup/cpu.max
100000 100000
$ docker exec web cat /sys/fs/cgroup/pids.max
100

Docker stores the same values, and docker inspect shows them under HostConfig: the fields Memory (bytes), NanoCpus (billionths of a core), and PidsLimit.

On a Linux host, you can find the container's group from the outside too. Where it lives depends on Docker's cgroup driver. With the systemd driver it's /sys/fs/cgroup/system.slice/docker-<full ID>.scope/, with the cgroupfs driver it's /sys/fs/cgroup/docker/<full ID>/. The driver is shown by docker info --format '{{.CgroupDriver}}'. On macOS and Windows, Docker Desktop keeps containers inside a Linux VM, so these directories aren't visible from the host; look from inside with docker exec.

When a process inside a group tries to take more memory than memory.max allows, the kernel runs the OOM killer and kills a process in that group. Neighbors on the host aren't affected: the limit acted locally. In Docker it looks like this: the container stopped with exit code 137 (128 plus the signal number for SIGKILL, 9), and docker inspect shows State.OOMKilled as true.

$ docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}}' web
137 true

The container itself writes nothing about the OOM. The process is killed by a signal and doesn't get to leave a log message. So you look for the cause in docker inspect, not in docker logs.

One more detail about memory. If only --memory is set, Docker allows the container the same amount of swap on top, so --memory 512m without --memory-swap on a host with swap gives up to 1 GB of memory and swap combined. To have no swap at all, set --memory-swap equal to --memory.

Trap one: namespaces and cgroups are the same thing

They get confused often because both take part in container isolation. The difference is the job each does. Take away namespaces, and the process sees every process and network on the host, but the memory limits stay. Without cgroups, it won't see anything extra, but it can take all the memory and CPU.

$ cat /proc/self/cgroup        # which cgroup am I in right now
0::/user.slice/user-1000.slice/session-2.scope
$ ls -l /proc/self/ns          # which namespaces do I have
lrwxrwxrwx net -> 'net:[4026531840]'
lrwxrwxrwx pid -> 'pid:[4026531836]'

One output is about resources, the other about visibility. These are two different kernel mechanisms running in parallel. They meet in one place: on cgroup v2, Docker gives a container its own cgroup namespace by default. That's why inside the container cat /proc/self/cgroup shows 0::/, as if the container's group were the root itself, while from outside it has a long path.

Trap two: a container won't bring down the host on memory

It will, if no limit is set. A container without --memory lands in a cgroup with memory.max equal to max, that is, no ceiling. A memory leak inside such a container keeps growing until the host's OOM killer starts killing processes, and neighboring containers or the host itself can take the hit.

docker run -d nginx:1.27.3-alpine                  # no limit: memory.max = max
docker run -d --memory 512m nginx:1.27.3-alpine    # with a limit: own group, own ceiling

Resource limits don't appear on their own. Whoever runs the container sets them, through flags or a manifest.

Trap three: remove the limit because it's causing the crash

The container crashed with OOMKilled: true, and the first instinct is to drop --memory. It will stop crashing, but the problem moves to the host: now its appetite is bounded only by the machine's memory. The right move is different. First figure out how much memory the app actually needs (memory.current while it's running, docker stats), then set a limit with some headroom. If consumption keeps growing forever, that's a leak, and the limit has nothing to do with it.

You can change the limit without recreating the container: docker update --memory works on both a running and a stopped container. If --memory-swap was set before, raise it in the same command, otherwise Docker refuses: memory can't be bigger than memory plus swap.

The container as the host sees it

One fact pulls the picture together. From the host OS's point of view, a container is an ordinary process. Its distinguishing features: it's placed in separate namespaces, its capabilities (the process's set of privileges) are cut down, and it's assigned to a cgroup with limits.

$ ps aux | grep nginx
root  4821  0.0  0.1 nginx: master process

On a Linux host, the container's process sits in ps right next to the rest of the host's processes, with a regular PID. On macOS and Windows it lives inside the Docker Desktop VM, and the host's ps won't show it. There's no magic: namespaces hide what's extra, cgroups limit resources, capabilities cut privileges.

What to remember

cgroups (control groups) limit the resources of a group of processes: CPU, memory, I/O, number of processes.

Limits live as files in /sys/fs/cgroup; the current version is cgroup v2. The root of the hierarchy has no limit files, child groups do.

docker run --memory 512m --cpus 1 --pids-limit 100 writes into memory.max, cpu.max, and pids.max of the container's group under the hood.

Hit memory.max, and the OOM killer shows up: exit code 137 and State.OOMKilled: true in docker inspect.

Namespaces are about visibility, cgroups are about resources. These are different mechanisms.

Without a limit, a container can take all the host's memory and trigger an OOM for its neighbors. Whoever runs the container sets the limit.

What you'll need for the tasks

• docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}}' <name>, fields HostConfig.Memory, HostConfig.MemorySwap, HostConfig.NanoCpus, HostConfig.PidsLimit
• docker exec <name> cat /sys/fs/cgroup/memory.max (and cpu.max, pids.max, memory.current)
• docker update --memory … --memory-swap … <name> and docker start <name>
• docker logs <name>
• see docker container update and Resource constraints

In the next lesson we'll cover capabilities: how a process's privileges get cut down, so root inside a container isn't root on the host.