Swap and the OOM killer

Why this matters

A service disappeared overnight, the application log is empty, not a line before it died. That's usually what the OOM killer's work looks like: the kernel killed the process with SIGKILL, and that signal has no handler, so the application never gets to write anything.

In this lesson we'll look at what the kernel does when memory runs out: how it frees page cache, when it reaches for swap, and what rules it uses to pick a victim. And where to look for the traces afterward.

How the kernel survives low memory

When free memory drops below a threshold (watermark), the kernel starts reclaim: it frees pages it can get back. The kswapd kernel thread does this in the background; you can see it on the host in ps.

The first candidates for eviction are page cache pages: they have a copy on disk. Anonymous pages (heap, stack) can't be dropped that way, so there's swap for them: the kernel writes an inactive page to disk and frees the physical memory.

swapon --show shows which swap devices are attached:

swapon --show

NAME      TYPE SIZE USED PRIO
/swapfile file   2G 340M   -2

If the output is empty, the machine has no swap.

Swap holds what hasn't been touched in a while: the heap of a dormant process, the memory of a daemon that wakes up once a day. It never becomes fast memory for programs. Every access to a swapped-out page costs disk I/O, so active data doesn't live in swap. If a system keeps reading from swap, it genuinely doesn't have enough memory.

swappiness

How eagerly the kernel swaps out anonymous pages instead of dropping cache is set by vm.swappiness:

sysctl vm.swappiness

vm.swappiness = 60

The range is 0 to 200, default 60. The lower the value, the less often the kernel touches swap. The upper bound was raised from 100 to 200 in kernel 5.8: values above 100 make sense when swap is faster than the disk holding your files, for example zram (swap in compressed memory).

For databases, swappiness is often lowered to 1-10: the database cares more about keeping its own data in RAM than the kernel does about preserving page cache. Don't set it to 0 without understanding why. At 0, the kernel leaves swap alone until it has cleared almost all the cache, and the system gets closer to OOM.

OOM killer

If memory has run out completely, reclaim didn't help, and there's nothing left to allocate, the kernel runs the OOM killer (out of memory). It sends one process SIGKILL to free memory and save the rest of the system.

SIGKILL can't be caught. The process dies immediately, and its parent sees exit code 137: 128 plus signal number 9. Code 137 in a supervisor's log or in docker ps -a is the first clue pointing to OOM.

Who gets killed: badness score

The victim is chosen by badness score. Roughly: the more memory a process holds (RSS, swap, page tables), the higher its score. The final score is visible in /proc/<PID>/oom_score, and you can shift it through /proc/<PID>/oom_score_adj, values from -1000 to +1000:

cat /proc/812/oom_score_adj

0

A positive adj makes a process a more attractive victim, +1000 means "kill this one first". A negative value protects it, -1000 effectively removes the process from the candidates:

echo -1000 > /proc/812/oom_score_adj

Any process can raise its own adj. It can only be lowered to the value last set by a process with CAP_SYS_RESOURCE (usually the starting value). Going lower requires that capability, otherwise the write returns Permission denied. A container with default permissions has no CAP_SYS_RESOURCE, so even root can't give itself a negative adj.

adj is inherited by children, including through exec. That's how you set adj for a service: a wrapper writes to /proc/self/oom_score_adj, then execs the program. systemd has the OOMScoreAdjust= unit setting for this, Docker gives you the docker run --oom-score-adj flag.

That's how critical services are protected. For example, sshd in OpenSSH on Linux sets its own oom_score_adj to -1000, and restores the original value for its sessions.

Where to find the trail

The OOM killer leaves a trace in the kernel log. Usually only root can read it:

sudo dmesg | grep -i "out of memory"

[8821.320993] Out of memory: Killed process 4172 (java) total-vm:6291456kB, anon-rss:3145728kB, ...

dmesg's buffer is a ring, so old lines get pushed out. The same kernel log, with history from past days, lives in journald. The -g flag searches by regular expression:

journalctl -k -g "Out of memory"

Check a previous boot with -b -1. This only works if the journal is stored on disk (there's a /var/log/journal directory):

journalctl -k -b -1 -g "Out of memory"

A classic on production: a java process with no limits ate all the memory, the OOM killer killed it overnight, and in the morning no one can figure out where the service went. These commands answer that question in ten seconds.

systemd-oomd

There's a second killer too. systemd-oomd is a userspace service: it watches memory pressure (PSI, pressure stall information) and kills a whole cgroup without waiting for the kernel. It's enabled by default in Fedora 34+ and Ubuntu Desktop 22.04+. Its work doesn't show up in the kernel log, you need to look in the service's own journal:

journalctl -u systemd-oomd

OOM in a container

A container with a memory limit (docker run --memory) lives in its own cgroup. When the cgroup hits its limit, the OOM killer fires inside it, even though the host still has memory free. It picks a victim among that cgroup's processes by the same badness and oom_score_adj rules. In the kernel log the line starts with Memory cgroup out of memory, so grep -i "out of memory" finds it too.

Keep in mind that the kernel computes /proc/<PID>/oom_score from the whole machine's memory, not from the cgroup's limit. In a container with a small limit, process scores end up nearly identical (say, 671 and 672). Compare the process size directly (the VmRSS line in /proc/<PID>/status) along with its oom_score_adj.

On Docker Desktop (macOS, Windows) the kernel lives inside an internal virtual machine, so you can't read its dmesg from your laptop. But cgroup v2 counts events on its own. Inside the container:

docker exec web cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.events

268435456
low 0
high 0
max 31
oom 2
oom_kill 2

memory.max is the limit in bytes (268435456 = 256 MB). In memory.events, the max counter shows how many times the cgroup hit its limit, and oom_kill shows how many processes inside it the OOM killer killed. The counters accumulate from when the cgroup was created.

If the container's main process was killed, Docker marks it in the container's state:

docker inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}}' web

true 137

If a supervisor is running inside the container and OOM killed its child process instead, the container stays Up. What's left is exit code 137 in the supervisor's log and oom_kill in memory.events.

A limit without swap is set like this: --memory 256m --memory-swap 256m. If you don't specify --memory-swap, Docker grants the container the same amount of swap on top (as long as the host has swap), and before OOM the process spends a long time dragging through swap first.

Common mistakes

• Looking for the cause of death in the application log. After SIGKILL the application writes nothing, you need to check the kernel log, memory.events, or exit code 137.
• Mistaking code 137 for an application bug. It's 128 + 9, the process was killed by a signal from outside.
• Setting vm.swappiness=0 "so it stops lagging". Swap doesn't get disabled, and the system reaches OOM faster.
• Protecting a service with nice. nice changes CPU priority and has no effect on the OOM victim choice. That's what oom_score_adj is for.
• Writing oom_score_adj into an already-running process and calling it done. The value disappears on the next restart, you need to set it where the process starts from.
• Checking only dmesg on a machine running systemd-oomd. oomd's victims don't show up in the kernel log.

What you'll need for the tasks

• Exit code 137 = 128 + 9 (SIGKILL); kill -l 9 prints the signal's name.
• docker logs <container>, docker inspect -f '{{.HostConfig.Memory}}' <container>.
• /sys/fs/cgroup/memory.max and /sys/fs/cgroup/memory.events inside the container.
• /proc/<PID>/oom_score and /proc/<PID>/oom_score_adj, ps to find the PID.
• The rule: any process can raise its own adj, lowering it below the starting value needs CAP_SYS_RESOURCE; adj is inherited through exec.