File systems and inodes

Why this matters

Three real-life admin situations all come down to one piece of file system design. You delete a file, and the space doesn't free up. There's space, but a new file can't be created. After a directory move, an app won't start even though all the files are right there. To troubleshoot these, you need to understand how a file's name differs from the file itself.

The name and the file live apart

In Linux, a file's name and its content are stored separately. The name lives in a directory. Everything else lives in a structure called the inode: size, permissions, owner, timestamps, and pointers to the data blocks on disk. There's no name inside the inode.

ls -i shows the inode number:

ls -i

1310721 app.log   1310733 config.yml

A directory works like a table mapping "name → inode number". When you open a file by name, the system looks up its inode in that table, then gets the data from the inode.

stat prints everything stored in the inode. The inode number and the count of names pointing at it are easy to check this way:

stat -c '%i %h %n' app.log

1310721 1 app.log

Here, 1310721 is the inode number, 1 is the link count (number of names), app.log is the name.

"There's space, but a file can't be created"

On ext2, ext3 and ext4, the number of inodes is fixed when the file system is created and doesn't grow afterward. Every file takes up one inode. If there are a lot of small files (caches, sessions, mail queues), inodes can run out before space does.

Not every file system works this way. XFS (the default on RHEL 9) and Btrfs allocate inodes on demand. XFS does have a ceiling: inodes can take up at most a given share of space (the -i maxpct= option of mkfs.xfs, 25% by default on partitions under 1 TB). In practice it's hard to hit. But ext4 is still the default on Debian and Ubuntu, so the failure below is a real one.

It looks strange: df -h shows free space, but creating a file fails with No space left on device. Inode usage is checked separately, with the -i flag:

df -i

Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/vda1     2560000 2560000     0  100% /

IUse% 100% means inodes are gone. The fix is deleting junk small files (old sessions, temp files), not adding more disk. For how to find the directory with too many files, see the lesson on disk space (du --inodes).

Hardlink

Since the name and the inode are separate, one inode can have several names attached to it. This extra name is called a hardlink (hard link):

ln file.txt hardlink.txt
ls -li file.txt hardlink.txt

1850139 -rw-r--r-- 2 root root 6 Oct  9 10:37 file.txt
1850139 -rw-r--r-- 2 root root 6 Oct  9 10:37 hardlink.txt

Both names share the same inode number, and the second column after the permissions shows 2: that's the link count. Neither name is "the real one"; they're equal.

Deleting one name doesn't touch the data. rm lowers the link count, and the data only disappears once the last name is deleted and the count hits zero (and no process still has the file open). You can find all the names of one inode with find:

find / -xdev -samefile file.txt

-xdev keeps find from crossing into other partitions: a hardlink can't point there anyway. Instead of -samefile, you can search by number: find / -xdev -inum 1850139.

Hardlinks have two limits, both following from how they work:

• they only work within a single file system, because an inode number only means something inside its own partition. Linking across partitions gives Invalid cross-device link;
• you can't hardlink a directory: ln: d: hard link not allowed for directory.

Symlink

A symlink (symbolic link) works differently. It's a separate file with its own inode, containing a path to the target.

ln -s /opt/app/releases/v2 /opt/app/current
ls -l /opt/app/current

lrwxrwxrwx 1 root root 20 Oct  9 10:37 /opt/app/current -> /opt/app/releases/v2

The l at the start of the line and the -> arrow show that it's a symlink and where it points. readlink prints the path stored inside the link, and readlink -f follows the whole chain and gives the final path.

A symlink points at a name, not at an inode. If the target gets deleted or renamed, the symlink becomes "broken": it's still there, but there's nothing you can open through it.

cat: sym: No such file or directory

find /opt/app -xtype l finds broken symlinks in a directory.

On the other hand, a symlink can point across file systems and at a directory. That's why app releases are often laid out in releases/<version> directories, with a current symlink pointing at the one in use: switching versions just means re-pointing one link.

You re-point a link like this:

ln -sfn /opt/app/releases/v3 /opt/app/current

-s creates a symlink, -f replaces an existing one, -n keeps ln from following the old link into the directory it points at. Without -n, running ln -sf on a symlink to a directory would create a new link inside that directory, and current would stay the same as before.

Common mistakes

• Deleted a file, but the space didn't come back. The file has a second name (a hardlink) somewhere else. Check the link count with ls -l or stat -c %h, find all the names with find -samefile.
• A relative symlink points at the wrong place. A relative path in a symlink is counted from the directory the link itself lives in, not from the directory you're standing in. ln -s b/t a/link creates a link to a/b/t. For a/link → b/t, you need ln -s ../b/t a/link.
• Moved a directory and everything broke. Symlinks with absolute paths to the old location became broken. After a move, check find <directory> -xtype l.
• ln -sf without -n didn't re-point current. A new link showed up inside the old release directory instead.
• Editing with sed -i "detached" a hardlink. sed -i writes a new file and swaps it in for the old one, so the file gets a new inode. The second name keeps pointing at the old content.
• df -h looks fine, but new files can't be created. Check df -i.

What you'll need for the tasks

• ls -li and stat -c '%i %h %n': the inode number and link count;
• find <where> -xdev -samefile <file> or -inum <number>: all the names of one inode;
• df -h <path>: how full a partition is;
• ls -l, readlink, readlink -f: where a symlink points;
• find <where> -xtype l: broken symlinks;
• ln -sfn <target> <link>: re-point a symlink.

All the tasks run inside a Docker container, a regular Ubuntu underneath. The labs don't touch your own system.

What to remember

A file's name lives in a directory, everything else (permissions, size, pointers to data) lives in the inode. A directory is a "name → inode" table.

On ext2/3/4 the number of inodes is fixed when the file system is created. When they run out, a new file can't be created, even though df -h shows space. Check df -i. XFS and Btrfs allocate inodes on demand.

A hardlink is just another name for the same inode, within one file system and not for directories. The data lives as long as at least one name does. A symlink is a separate file with a path to a target: it can point anywhere, and it can end up broken.

The next lesson covers file descriptors: how processes hold files open, and why those can run out too.

For more on how hardlinks and symlinks work at the inode level, see the lesson "Hard and symbolic links" in the "Linux: the file system in depth" track.