Pseudo-filesystems: /proc and /sys

Why

On a minimal server or in a container you often do not have ps, free, ip or lscpu installed. But /proc and /sys are almost always there. These utilities get their data from those two directories, so once you know how to read them you get the same information about processes, memory and the network without the tools.

Files that do not exist on disk

A regular file lives on a partition, takes up blocks, and has an inode with metadata. With /proc and /sys it works differently.

/proc is a virtual filesystem with information about processes and the kernel. /sys is a virtual filesystem about devices, drivers and kernel parameters. The word "virtual" here is literal: there is no content on disk, the kernel generates it on the fly when you read it.

Easy to check. Both are mounted as separate filesystem types, not as a disk partition:

mount | grep -E ' /proc | /sys '

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

The types proc and sysfs are kernel interfaces, not a filesystem on a partition like ext4. Hence the first gotcha: /proc takes up no space on disk. The kernel assembles the answer again on every access.

In a Docker container the output looks almost the same, but /sys is mounted read-only: you will see sysfs on /sys type sysfs (ro,...).

What is in /proc

Some paths in /proc are numbers. Each number is a process PID, and inside that directory the kernel shows its state. For example, /proc/<PID>/status gives the process's memory and status. The file is long, so let's pick out the lines we need:

grep -E '^(Name|State|Pid|PPid|VmRSS|Threads):' /proc/self/status

Name:   grep
State:  R (running)
Pid:    4821
PPid:   4630
VmRSS:      2304 kB
Threads:        1

VmRSS is how much physical memory the process actually occupies right now. /proc/self always points to the process reading the file, which is handy for checking without substituting a specific PID. Here grep is reading the file, so Name: grep. Besides status, a process directory also has cmdline (the launch command, arguments separated by a null byte) and fd (open files, more on that below).

Separate files outside the numeric directories describe the system as a whole. /proc/meminfo is the machine's memory, /proc/cpuinfo is its processors:

grep -E 'model name|processor' /proc/cpuinfo | head -n 4

processor       : 0
model name      : AMD Ryzen 7 5800X 8-Core Processor
processor       : 1
model name      : AMD Ryzen 7 5800X 8-Core Processor

The second gotcha hides here. /proc/cpuinfo looks like a text file, but it is an interface to the kernel. The kernel forms its numbers at read time; there is no such file on disk. cat just pokes the kernel and prints the answer. On ARM processors the model name line may be missing, but processor is always there.

What is in /sys

/sys is organized by device and subsystem. Network interfaces show up in /sys/class/net, block devices (disks) in /sys/block:

ls /sys/class/net

enp3s0  lo  wlan0

ls /sys/block

nvme0n1  sda

Going further in: /sys/class/net/enp3s0/address shows the interface's MAC address, /sys/class/net/enp3s0/mtu its MTU, /sys/block/sda/size the disk size in 512-byte sectors (even if the disk's own sector is 4096). The kernel returns these values from its own structures, not from a file on a partition.

One more useful branch: /sys/fs/cgroup. Through it the kernel exposes control groups (cgroups), which limit process resources. Docker puts a container into its own group, and inside the container on cgroup v2 the memory limit shows up in /sys/fs/cgroup/memory.max:

cat /sys/fs/cgroup/memory.max

67108864

That is 64 MB: the container was started with --memory 64m. If there is no limit, the file reads max.

Files you can write to

Some paths are writable, and writing to them changes kernel behavior. The /proc/sys branch is the same set of parameters that sysctl configures: the dots in the parameter name correspond to slashes in the path. Only root can write there:

cat /proc/sys/net/ipv4/ip_forward

0

echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

sysctl net.ipv4.ip_forward

net.ipv4.ip_forward = 1

The first command showed that packet forwarding is off, the second turned it on, the third read the same value through sysctl. sudo echo 1 > ... will not work: the > redirection is done by your shell without root privileges, which is why tee is used here.

Writing to a file like this does not save bytes to disk. It passes the value to the kernel, and the kernel starts behaving differently right away. After a reboot the value goes back to its default unless you write it to a config file.

People used to put everything in /etc/sysctl.conf. These days the systemd documentation recommends separate drop-in files in /etc/sysctl.d/, and on some distributions /etc/sysctl.conf is no longer read at all:

echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/99-local.conf

sudo sysctl --system

* Applying /usr/lib/sysctl.d/50-default.conf ...
* Applying /etc/sysctl.d/99-local.conf ...
net.ipv4.ip_forward = 1

sysctl --system applies everything from the sysctl.d directories. Files from all the directories are sorted by name and applied in order, so a file with a higher number overrides the distribution's defaults. The output shows which files were picked up and what got applied. Your list of files will be different; it depends on the distribution.

Related facts for context

Inside /proc/<PID>/fd you can see the process's open file descriptors. A file descriptor is a per-process non-negative integer the kernel uses to address an open file. Each entry in fd is a symbolic link to an open file. Descriptors 0, 1, 2 are stdin, stdout and stderr:

ls -l /proc/self/fd

total 0
lrwx------ 1 user user 64 Oct 12 10:00 0 -> /dev/pts/2
lrwx------ 1 user user 64 Oct 12 10:00 1 -> /dev/pts/2
lrwx------ 1 user user 64 Oct 12 10:00 2 -> /dev/pts/2
lr-x------ 1 user user 64 Oct 12 10:00 3 -> /proc/4825/fd

Here 0, 1 and 2 lead to the terminal. Descriptor 3 was opened by ls itself: it is the directory it is currently reading. For another process you can see the same way which logs and sockets it writes to.

Through /sys/block you can see disks, and there is often a RAID array behind them. RAID levels differ in failure margin and capacity lost:

Level	Minimum disks	Usable capacity	Survives a failure of
RAID 0	2	100%	0 disks
RAID 1	2	50% (mirror)	1 disk
RAID 5	3	(n-1)/n	1 disk
RAID 10	4	50%	depends on the pair

We cover RAID in detail in a separate lesson in this module.

One more detail about paths in the system. The split between /bin and /usr/bin historically meant, per the spec, "critical for boot" versus "application". On modern distributions this split has been blurred by the usrmerge process: /bin is now usually a symlink to /usr/bin:

ls -ld /bin

lrwxrwxrwx 1 root root 7 Apr 22  2024 /bin -> usr/bin

Common mistakes

Treating /proc/meminfo and /proc/cpuinfo as container limits. Inside a container these files show the whole machine (in Docker Desktop, its virtual machine), not --memory and --cpus. That is why free lies about available memory inside a container too. Check the limit in /sys/fs/cgroup/memory.max and /sys/fs/cgroup/cpu.max.

Writing to /proc/sys inside a container. Docker mounts /proc/sys read-only, and echo 1 > /proc/sys/net/ipv4/ip_forward ends with Read-only file system even as root. Set the container's network namespace parameters at startup: docker run --sysctl net.ipv4.ip_forward=1 ....

Expecting sysctl -w or a write to /proc/sys to survive a reboot. It will not. You need a file in /etc/sysctl.d/.

Looking in /proc/<PID> after the process has exited. The directory disappears along with the process. /proc/self is different for every reader, so ls /proc/self from your own shell will not show another process's state.

What you'll need for the tasks

• ls /proc and the /proc/[0-9]* pattern: a list of PIDs without ps.
• /proc/<PID>/status (the Name, PPid, VmRSS lines) and /proc/<PID>/cmdline.
• ls -l /proc/<PID>/fd and readlink: which files a process has open.
• /proc/cpuinfo (processor lines), /proc/meminfo (MemTotal).
• /sys/class/net/<interface>/address and mtu, /sys/fs/cgroup/memory.max.
• /proc/sys/... and sysctl: reading kernel parameters.
• grep, awk '{print $2}' and a for loop over files.

The next lesson covers tmpfs: a filesystem that lives in RAM.