Anonymous memory, page cache, RSS

How anonymous memory differs from file-backed memory, what page cache is, why free shows little free memory, and why a process's RSS is misleading.

Why this matters

On a server, free shows 300 MiB free out of 8 GiB, and the urge is to restart something right away. Usually that's unnecessary: the memory is held by file cache, and the kernel will hand it back the moment something asks for it. The opposite mistake happens too. A process with a large RSS looks like the main memory hog, even though most of that RSS is shared with other processes or backed by a file on disk. To avoid restarting the wrong service, you need to tell the kinds of pages apart.

Two kinds of pages

A process's memory pages split into two kinds based on one thing: whether there's a file behind the page.

File-backed memory is a mapping of a file: program code, libraries, a file opened through mmap. A page like this has a copy on disk.

Anonymous memory isn't backed by a file. It's the process's heap and stack, data the program computed while running. None of it exists on disk anywhere.

The difference matters when memory runs short. The kernel can simply drop a clean file-backed page: the data is in the file, and the kernel will read it again on the next access. It writes a modified (dirty) page back to the file first. An anonymous page has nowhere to go; it can only be pushed out to swap.

Page cache

When a process reads a file, the kernel puts the pages it read into page cache and keeps them in memory. Reading the same file again skips the disk. Writes go through the cache too: first into memory, then to disk in the background.

That's where the newcomer's big surprise with free comes from:

free -h

               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       1.9Gi       300Mi        12Mi       5.6Gi       5.5Gi
Swap:          2.0Gi          0B       2.0Gi

300 MiB in the free column looks alarming, but it's normal. 5.6 GiB is taken up by page cache (the buff/cache column), and the kernel will give that memory back as soon as processes need it: clean cache pages can be freed without writing to disk.

The column to watch is available. It's an estimate of how much memory can be handed to new processes without touching swap. The kernel calculates it from free memory, the part of the caches that can be released, and the watermarks (thresholds below which free memory won't drop). The number itself lives in /proc/meminfo, in the MemAvailable line:

grep -E 'MemTotal|MemFree|MemAvailable|^Cached' /proc/meminfo

MemTotal:        8131204 kB
MemFree:          307512 kB
MemAvailable:    5767340 kB
Cached:          5602184 kB

These numbers are just an example; yours will be different.

Inside a container, free and /proc/meminfo show the whole machine's memory, not the container's limit. A container sees its own memory in the cgroup file /sys/fs/cgroup/memory.stat (cgroup v2): the anon line is anonymous memory, the file line is page cache along with tmpfs and shmem. Both count toward the container's limit. When the limit is reached, the kernel evicts cache first, and only when there's nothing left to free does the OOM killer kick in.

Shared memory and RSS

A page can belong to several processes at once. The libc library is loaded into memory once but mapped into the address spaces of hundreds of processes. That's shared memory. A separate kind of it is shmem: pages of tmpfs, /dev/shm, and shared anonymous mappings.

This is why RSS is misleading. In /proc/<PID>/status the kernel breaks it down into parts:

grep -E 'VmSize|VmRSS|Rss' /proc/812/status

VmSize:   720160 kB
VmRSS:     98400 kB
RssAnon:   61200 kB
RssFile:   35800 kB
RssShmem:   1400 kB

Replace 812 with your own process's PID.

• RssAnon: the process's anonymous pages (heap, stack).
• RssFile: pages of mapped files, including program code and libraries.
• RssShmem: shared memory (tmpfs, /dev/shm, shared anonymous mappings).

VmRSS = RssAnon + RssFile + RssShmem. Add up the RSS of every process and you get more than the actual usage, sometimes more than the physical memory itself: shared pages get counted many times over. The figure for "what this specific process actually eats" is closer to RssAnon. More precise still is PSS (proportional set size) from /proc/<PID>/smaps_rollup: each shared page is split evenly between the processes holding it.

Common mistakes

Panicking over a small free. Memory held by cache is free for processes to take. A shortage shows up as a small available, not a small free.

Treating page cache as a leak. The cache grows as long as there's free memory, and shrinks under pressure. There's no reason to drop it by hand on a production server; the next reads will just come from disk.

Hunting for the culprit by the biggest RSS. A process that mapped a large file or holds a shared /dev/shm segment will show a big RSS, even though it may have few anonymous pages of its own. Compare RssAnon instead.

Adding up the RSS of every worker. Shared pages land in the sum once for every process holding them.

Checking free inside a container. That shows the host's memory. The container's limit and usage live in /sys/fs/cgroup/memory.max and /sys/fs/cgroup/memory.stat.

What you'll need for the tasks

• /proc/<PID>/status, the VmRSS, RssAnon, RssFile, RssShmem lines (units in kB).
• The loop for d in /proc/[0-9]* and /proc/<PID>/cmdline: listing processes without ps.
• /sys/fs/cgroup/memory.current and memory.max: how much memory the cgroup is using and its limit (in bytes).
• /sys/fs/cgroup/memory.stat, the anon and file lines (in bytes).
• kill <PID>: stop a process.
• dd to write a file of a given size.
• docker exec to run commands inside the lab environment's container.

The next lesson covers swap, swappiness, and the OOM killer.