Disk: df, du, mount, inode

"No space left on the disk"

The most common server incident: space runs out, and the service stops. The database won't write, the app crashes with No space left on device, logs cut off. Untangling it takes two commands: df (how much is used on each partition) and du (who's using it).

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   38G  512M  99% /
/dev/vdb1       100G   12G   83G  13% /var/lib/postgresql
tmpfs           2.0G     0  2.0G   0% /run

df -h shows partitions and how full they are in human-readable form (-h means K, M, G instead of 1-kilobyte blocks). The Use% column tells you right away where the trouble is: the root / is 99% full.

df vs. du

These are different tools, and people mix them up.

• df looks at filesystems: total, used and free space on each partition. It answers "which partition is full".
• du adds up directory sizes: how much the files inside them take up. It answers "what exactly ate the space".

The combo works like this: df shows that / is full, then du finds the culprit.

du -sh /var/* | sort -h | tail -5

1.2G    /var/cache
3.4G    /var/lib
12G     /var/log

du -sh /var/* gives the size of each subdirectory (-s for one line per directory, -h in human-readable form). sort -h sorts taking the G, M, K suffixes into account, and tail -5 keeps the five biggest. The culprit here is /var/log at 12 gigabytes. You repeat the same trick one level down, du -sh /var/log/* | sort -h | tail -5, until you reach specific files.

Without sudo, du can't enter directories closed off from your user, prints Permission denied, and counts less than is actually used. On a server, run sudo du ....

Mount points

A partition becomes accessible once it's mounted at some point in the directory tree. The Mounted on column in df's output is exactly that: mount points. In the example above, /var/lib/postgresql is mounted from a separate disk, /dev/vdb1: the database data doesn't live on the root partition.

This matters when you're troubleshooting. Space can run out on / while /var/lib/postgresql still has plenty: they're physically different disks. df -h <path> shows which partition a specific path lives on.

df -h /var/lib/postgresql/16/main

Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb1       100G   12G   83G  13% /var/lib/postgresql

lsblk shows the tree of disks and their partitions along with mount points. You can pick which columns to show with -o:

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT

NAME    SIZE TYPE MOUNTPOINT
vda      40G disk
└─vda1   40G part /
vdb     100G disk
└─vdb1  100G part /var/lib/postgresql

You can see the physical disks (vda, vdb), their partitions (vda1, vdb1), and where each one is mounted. The mount command with no arguments prints all mounted filesystems along with their mount options (ro, noexec, size=..., and others).

One more detail in the same area. On ext4, 5% of blocks are reserved for root by default, so in df, Used and Avail add up to less than Size. A regular user hits "disk full" sooner than root does.

When df and du disagree

It happens: df says "full", but du on the same partition shows little used. A common reason is that a file was deleted while still open. A process holds a file open, someone deletes it (rm), and the space doesn't come back. The file has no name anymore, but the data stays on disk until the last process closes the descriptor. The typical case: someone deletes a bloated log by hand, while the app keeps writing to it.

Finding such files:

sudo lsof +L1

COMMAND PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK NODE NAME
nginx   812 www    5w   REG  253,1  4294967296     0 1311 /var/log/nginx/access.log (deleted)

+L1 picks out open files with fewer than one link (the NLINK column equal to 0), that is, deleted but still open. The PID column has the process number holding the file, and SIZE/OFF shows how much space it takes up. You need root here: without it, lsof can't look into other users' processes.

In minimal images and containers, lsof is often not installed. The same descriptors are then visible straight from /proc: every process has a /proc/<PID>/fd directory, with a link for each open file. A link to a deleted file ends in (deleted).

sudo find /proc/[0-9]*/fd -lname '*(deleted)' 2>/dev/null

/proc/812/fd/5

The number after /proc/ is the PID. ps -o pid,cmd -p 812 shows which process it is, and ls -l /proc/812/fd/5 shows which file it's holding.

The space comes back once the process closes the file. Usually that means restarting the process (sudo systemctl restart nginx). The file itself won't come back, but the process will open a new one at the same path and write there.

A second cause of the mismatch: du can't see files under a mount point that has another partition mounted on top of it. The files still sit in the old directory and take up space on the parent partition, but while something else is mounted on top, neither du nor ls can see them.

Space is free, but files can't be created: inodes

Every file and directory uses up two resources on a partition: space for the data and an inode, a record about the file in the filesystem's internal table, holding the owner, permissions, size and a pointer to the actual data. The number of inodes is set when the filesystem is created (on ext4, with the mkfs.ext4 command) and doesn't grow afterward. If a partition has millions of small files (sessions, cache, queued emails), inodes run out before space does. A new file can't be created, with the same No space left on device error, while df -h still shows free gigabytes.

df -i shows how full the partition is by inode count:

df -i /var

Filesystem      Inodes   IUsed IFree IUse% Mounted on
/dev/vda1      2621440 2621440     0  100% /

IUse% at 100%: no more files can be created. du --inodes (in GNU du, that is, on Linux and WSL2) counts where most of them are:

sudo du --inodes -s /var/* | sort -n | tail -5

1843    /var/cache
21950   /var/lib
2597410 /var/spool

This kind of clutter is usually cleaned up by age, with find. For example, log archives older than 30 days:

sudo find /var/log/app -name '*.gz' -mtime +30 -delete

-mtime +30 means "modified more than 30 full days ago". A file exactly 30 days and some hours old won't match: find rounds the age down to whole days. Before -delete, it's worth running the same command without it first, to see what would get deleted.

Common mistakes

• Only checking df -h. It shows space in bytes. If you see No space left on device with space available, check df -i.
• Deleting a big log with rm and expecting the space back. As long as a process holds the file open, the space stays used. A log a process is actively writing to gets cleared with truncate -s 0 file or by rotation (logrotate), not rm.
• Restarting the wrong process. Descriptors are inherited by child processes. If both a parent and its children hold the file, the space comes back only once all of them are closed. lsof +L1 and find /proc show every PID involved.
• Cleaning up by eye. Running rm -rf on a directory that "doesn't look needed", without du or looking at what's in it. Find what's taking up space first, then decide what's safe to delete.
• Mixing up partitions. Space ran out on /, but you clean up /home, which is mounted from a different disk. df -h <path> tells you which partition a directory lives on.

What you'll need for the tasks

The labs run in an ubuntu:24.04 container, where you're root, so you don't need sudo. The partition you'll be working on is mounted at /srv/data.

• df -h <path> and df -i <path>: how full a partition is by space and by inodes.
• du -sh <directory>/* | sort -h and du --inodes -s <directory>/* | sort -n: who's using the space and where the files pile up. Repeat one level down until you find the culprit.
• find <directory> -mtime +N without -delete to check, and with -delete to remove.
• find /proc/[0-9]*/fd -lname '*(deleted)', ps -o pid,cmd -p <PID>, ls -l /proc/<PID>/fd: who's holding a deleted file.
• ps aux and kill <PID>: find and stop a process.

What to remember

df shows how full partitions are ("where it's full"), du shows directory sizes ("what ate it"). Troubleshooting goes from df to du.

Partitions are mounted at points in the tree, and space can run out on one partition while a neighboring one has plenty. df -h <path> tells you which partition you're dealing with.

If df and du disagree, look for deleted open files (sudo lsof +L1, or without lsof, sudo find /proc/[0-9]*/fd -lname '*(deleted)'). The space comes back once the process restarts.

No space left on device with space available means inodes ran out. Check df -i.

A full disk often means old logs, dumps or temp files were never cleaned up. The fix is rotation and scheduled cleanup, not a one-off rm.