The Linux filesystem

You've logged into someone else's server, and you need to find the nginx config, figure out what's filling up the disk, and see where the database keeps its data. If you know how the directory tree is laid out, that's three commands. If you don't, it's half an hour of find / by guesswork. This lesson is about the map: which folder is responsible for what, and where to look first.

Everything is a file

In Linux, the same system calls (open, read, write) work with different kinds of things:

• regular files and directories;
• block devices, meaning disks: /dev/sda, /dev/nvme0n1;
• character devices: /dev/null, /dev/random;
• named pipes and Unix sockets: they have a name in the directory tree;
• information about the kernel and processes in /proc and /sys.

In practice this means cat, ls, grep, and redirections work with almost everything in the system. For example, cat /proc/cpuinfo reads CPU information the same way as a regular text file, even though no such file exists on disk.

Network sockets (TCP, UDP) have no name in the directory tree. This is the one place where the "everything is a file" rule stops being literally true.

FHS: where things live

FHS (Filesystem Hierarchy Standard) is an agreement about which folder is for what. Nearly every distribution follows it. Here's what the root / looks like on a typical Ubuntu server:

/
├── bin -> usr/bin     symlink: executable files
├── boot               kernel and initramfs
├── dev                devices: /dev/sda, /dev/null, /dev/tty
├── etc                configs: nginx/nginx.conf, ssh/sshd_config, hosts
│   ├── nginx/
│   ├── ssh/
│   └── systemd/
├── home
│   └── deploy/        home directory of the user deploy
├── lib -> usr/lib     symlink: libraries
├── opt                large third-party apps, each as one self-contained folder
├── proc               virtual filesystem: /proc/cpuinfo, /proc/<pid>/
├── root               root's home directory (not /home/root)
├── run                sockets and pid files of running services, gone after reboot
├── sbin -> usr/sbin   symlink: admin programs
├── srv                data the server serves out (often empty)
├── sys                virtual filesystem about hardware and the kernel
├── tmp                temporary files
├── usr
│   ├── bin/           programs from packages
│   ├── local/         software installed by hand, not by the package manager
│   └── sbin/          admin programs from packages
└── var
    ├── cache/         package manager and app caches
    ├── lib/           service data: postgresql, docker
    └── log/           logs: syslog, nginx/, auth.log, journal/

Symlinks /bin, /sbin, /lib point into /usr on Ubuntu, Debian, and Fedora (the so-called merged /usr). On Alpine 3.20 these are still plain folders. It makes no difference in practice: /bin/sh is found either way.

A cheat sheet for the folders you'll need most often:

Path	What's there
/etc	configs for the system and services
/var	data that changes: logs, caches, queues, databases
/var/log	logs (syslog, nginx/, auth.log)
/var/lib/<service>	a service's working data: database files, state
/usr/bin, /usr/sbin	programs installed by the package manager
/usr/local	software installed by hand: /usr/local/bin for programs
/opt	third-party apps, each in its own folder: /opt/<app>
/home/<user>	a user's home directory
/root	root's home directory
/tmp	temporary files, may be gone after reboot
/run	pid files and sockets of running services
/proc, /sys	virtual filesystems with info about the kernel and processes
/dev	devices

Something broke on the server? First check /etc/<service> (how it's configured) and /var/log (what happened). If the question is about data, go to /var/lib/<service>.

/usr/local vs /usr/bin

The package manager (apt, dnf, apk) puts files into /usr/bin. If you put your own script there by hand, the next package update might overwrite or remove it. Your own programs and scripts go into /usr/local/bin: the package manager doesn't touch that folder, and it usually comes earlier in PATH than /usr/bin.

echo $PATH

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

/opt is for applications that arrive whole, with their own libraries and configs: for example, /opt/<app>/bin/<program>. The system won't find such a program by name until its folder is added to PATH or a symlink is made in /usr/local/bin.

/tmp

A program shouldn't assume that a file in /tmp will survive until the next run. Exactly when /tmp gets cleared depends on the distribution: sometimes it's tmpfs in memory and everything disappears on reboot, sometimes systemd-tmpfiles deletes files on a timer. Don't keep anything important in /tmp.

/proc and /sys

/proc and /sys don't live on disk. The kernel assembles their content at the moment you read it. /proc/<pid>/ shows the process with that number, /proc/self/ shows the process that's currently reading.

ls -l /proc/self/exe

lrwxrwxrwx 1 root root 0 Oct  9 06:59 /proc/self/exe -> /usr/bin/ls

ls read the link to its own executable. Inside a container /proc exists too, but /proc/meminfo there shows the memory of the whole host, not the container's limit. That's a common trap when you try to find out how much memory is available from inside.

Commands for every day

• pwd — where am I right now.
• ls -la — the whole contents of a directory, including hidden files, with permissions and owner.
• cd - — go back to the previous directory.
• tree -L 2 — a tree two levels deep. On Ubuntu it's installed separately (apt install tree); on Alpine it's already there.
• du -sh <path> — how much space a directory takes up as a whole.
• df -h — how much free space is on each partition.
• cat /etc/os-release — which distribution and which version.

Here's ls -la /etc in the ubuntu:24.04 container (start of the output):

total 296
drwxr-xr-x 1 root root    4096 Oct  9 06:59 .
drwxr-xr-x 1 root root    4096 Oct  9 06:59 ..
-rw------- 1 root root       0 Sep 17 02:10 .pwd.lock
drwxr-xr-x 2 root root    4096 Sep 17 02:17 alternatives
drwxr-xr-x 8 root root    4096 Sep 17 02:10 apt
-rw-r--r-- 1 root root    2319 Mar 31  2024 bash.bashrc

The dot . means the directory itself, .. the parent. .pwd.lock is hidden: its name starts with a dot, so without -a you won't see it. The -A flag shows the same thing but without . and ... The first letter of the line tells you the type: d directory, - regular file, l symlink.

du -sh with * shows the size of each item in a folder separately. That's how you find the log that has grown too big:

du -sh /var/log/*

8.0K	/var/log/alternatives.log
32K	/var/log/apt
60K	/var/log/bootstrap.log
184K	/var/log/dpkg.log
...

Services' Unix sockets are visible in the directory tree too. For example, through /var/run/docker.sock (which is the same as /run/docker.sock, because /var/run has long been a symlink to /run) the docker client talks to the daemon.

Common mistakes

• Looking for a config in /var or logs in /etc. Settings go in /etc, anything that changes while running goes in /var.
• Putting your own scripts in /usr/bin. That's the package manager's territory; your own software goes in /usr/local/bin or in /opt/<app>.
• Storing anything important in /tmp. It gets cleared on reboot or on a timer.
• Writing /home/root. root's home directory is at /root.
• Treating /proc as a regular folder: copying it into a backup or looking there for files on disk. Its content exists only while the system is running.
• Running du -sh /var/log and not seeing which file is big. The total for the folder comes from -s; for each item you need du -sh /var/log/* or du -a /var/log | sort -n | tail.

What you'll need for the tasks

• Enter a container with docker exec -it <name> sh and leave it with exit.
• The "Path / What's there" table from this lesson: /etc, /var/log, /var/lib, /usr/local/bin.
• ls -la, cat, du -sh <path>/*, echo $PATH.
• mkdir -p creates a folder along with any missing parents, cp copies a file: cp <from> <to>.