File descriptors and /proc/<PID>/fd

Why this matters

A service ran fine for a week and then started crashing with Too many open files. Or the error log is empty, but there are definitely errors. Or you can't touch a file because "something is holding it". All three questions come down to one tool: the list of a process's open descriptors in /proc/<PID>/fd. There you see what the process has actually opened, without touching the code or configs.

What a file descriptor is

When a process opens a file, the kernel hands it a file descriptor (fd). It's a small non-negative number the process then uses to refer to the open file. A descriptor is the process's reference to an open resource, not the file itself.

Every process has three descriptors from the start:

• 0 is stdin, standard input;
• 1 is stdout, standard output;
• 2 is stderr, the error stream.

Then come 3, 4 and so on: files, network connections (sockets) and pipes the process opened while running. To the kernel these are all open files, so one mechanism covers every case.

The kernel assigns the lowest free number to a new descriptor. A process opens 3 and 4, closes 3, and the next file it opens gets 3 again.

/proc//fd

A process's open descriptors are visible in the /proc/<PID>/fd/ directory. Each descriptor there is a symbolic link to whatever is open:

ls -l /proc/812/fd

total 0
lr-x------ 1 root root 64 Oct  9 16:07 0 -> /dev/null
l-wx------ 1 root root 64 Oct  9 16:07 1 -> /var/log/app/app.log
l-wx------ 1 root root 64 Oct  9 16:07 2 -> /var/log/app/app.log
lrwx------ 1 root root 64 Oct  9 16:07 3 -> socket:[48213]
lrwx------ 1 root root 64 Oct  9 16:07 4 -> /var/lib/app/data.db

What you can see here:

• stdout (1) and stderr (2) are redirected to the same log, stdin (0) points to /dev/null;
• 3 is a socket: a network connection has no path, so instead you get socket:[number], where the number is the socket's inode number;
• 4 is the database file, opened for reading and writing.

The link's permissions show the open mode: lr-x is read-only, l-wx is write-only, lrwx is read-write. A pipe between processes looks like pipe:[number], and the number matches on both ends of the same pipe.

Count how many descriptors a process holds:

ls /proc/812/fd | wc -l

5

To look at other users' processes, you need root. A regular user only sees the descriptors of their own processes, and gets Permission denied for anyone else's. That's why the commands above typically run through sudo on a server.

The reverse question, "which process holds this file", is answered by searching the links of every process at once:

sudo find /proc/[0-9]*/fd -lname /var/lib/app/data.db 2>/dev/null

/proc/812/fd/4

The process number sits in the path between /proc/ and /fd. lsof (from the disk lesson) can do the same, but it might not be installed on a server, while /proc is always there.

"too many open files"

Every process has a limit on the number of open descriptors. You can see it in /proc/<PID>/limits:

grep 'Max open files' /proc/812/limits

Max open files            1048576              1048576              files

The first number is the soft limit, the second the hard limit. These are the defaults in a Docker container; on your machine and on a server they'll be different. We'll cover where they come from and how to change them in the next lesson.

When a program opens files or connections and forgets to close them, descriptors pile up. This is a descriptor leak. At some point the process hits the limit, and every next attempt to open a file or connection fails with the error EMFILE (errno 24). The error text depends on the program:

bash: line 4: /tmp/f: Too many open files

Error: EMFILE: too many open files, open '/etc/hostname'

The first line is from bash on Ubuntu, the second from Node.js. On Alpine images (which use the musl library instead of glibc), the same errno 24 reads differently: Python there prints OSError: [Errno 24] No file descriptors available. Search by the number and the EMFILE code, not by the text.

This is a common failure in network services: every connection is a descriptor, and if connections aren't closed, they pile up. The diagnosis is simple: watch how a process's descriptor count changes.

ls /proc/812/fd | wc -l

Repeat the command a few times with a pause of a few seconds. If the number keeps growing and never comes back down, it's a leak. Next, look at ls -l /proc/812/fd: if dozens of descriptors point to the same file, or they're all socket:[...], you can see which resource isn't being closed. The bug is in the code that isn't closing resources, and that's where you fix it.

Connection to inodes

This is where what came before ties together. A deleted but still open file (from the disk lesson) is a case where the name was removed from the directory, but the process still holds a descriptor to the inode. The data lives on as long as the descriptor is alive. That's why du doesn't see the file, but the space is still used. In /proc/<PID>/fd such a link ends with (deleted).

Common mistakes

• Raising the limit instead of finding the leak. A limit ten times higher pushes the crash back, but doesn't remove it: the leak keeps growing while the process is alive.
• Fixing a leak by restarting. The counter resets, the problem comes back after the same amount of time, and the work in progress at the moment of the crash is lost.
• Counting descriptors with ls /proc/self/fd. self is the process of the ls command itself, and the output will also include the directory ls opened to read it. Look at the specific PID instead.
• Looking for errors only in a file named error.log. Where a process writes its error stream is decided by whoever started it. The link /proc/<PID>/fd/2 shows the truth.
• Looking at the descriptors of the wrong process. A service can have a supervisor parent and child processes; the leak belongs to whichever one opens the files, not necessarily the one that started it.

What you'll need for the tasks

The labs run in containers, you're root in them, no sudo needed.

• ps or ps -eo pid,ppid,args: find the PID of the process you need.
• ls -l /proc/<PID>/fd: where each descriptor points.
• ls -l /proc/<PID>/fd | grep -c socket and ls /proc/<PID>/fd | wc -l: count sockets and all descriptors.
• find /proc/[0-9]*/fd -lname <path>: who holds the file. On Alpine, find is from BusyBox and has no -lname flag; there you go through ls -l /proc/<PID>/fd for each process instead.
• Measuring "twice with a pause" to tell a leak apart from a normal descriptor count.
• In the second lab the service is written in bash. How bash opens and closes descriptors is described in the bash manual, section Redirections.

What to remember

A file descriptor is a number a process uses to refer to an open file, socket or pipe. 0, 1, 2 are stdin, stdout, stderr, and then come the open resources.

A process's open descriptors are visible in /proc/<PID>/fd/. A number that keeps growing and never comes back down means a leak; sooner or later the process will hit "too many open files".

A deleted open file is kept alive by its descriptor: the space isn't freed while the descriptor is alive.

In the next lesson we'll cover process limits (ulimit), including the limit on these very descriptors.