Observing processes: ps, lsof, strace, /proc

Why

A service is acting up: it won't start, it's silent, it's hung. A restart sometimes helps, but it doesn't explain the cause. To understand what's going on, you look at a process from three angles: which processes are running at all, which files and connections they hold, which system calls they're making right now. Each question has its own tool.

ps gives you a snapshot. lsof shows open files and sockets. strace listens in on system calls. Below is one example of each, and at the end the same result straight from the kernel, through /proc.

ps: a process snapshot

ps aux prints every process in the system at once. The output is a table with fixed columns:

$ ps aux
USER   PID %CPU %MEM    VSZ   RSS TTY  STAT START  TIME COMMAND
root     1  0.0  0.1 168240 11500 ?    Ss   09:00  0:02 /sbin/init
www    812  1.3  2.4 720160 98400 ?    Sl   09:01  0:21 nginx: worker

What the columns mean:

Column	Meaning
USER	process owner
PID	process number
%CPU	share of the processor
%MEM	share of physical memory
VSZ	virtual memory in KB: everything the process has reserved
RSS	physical memory actually in use, in KB (Resident Set Size)
TTY	controlling terminal, ? if there is none
STAT	state (R, S, D, Z, T) and flags
COMMAND	what's running, with arguments

One trap: plain ps aux shows processes, not the threads inside them. A multithreaded service looks like a single line, even though a dozen threads are spinning inside it. The -L flag pulls out the threads:

ps -L -p 812      # each thread of process 812 on its own line (LWP column)

    PID     LWP TTY          TIME CMD
    812     812 ?        00:00:00 python3
    812     814 ?        00:00:00 python3
    812     815 ?        00:00:00 python3

If you only need specific columns, list them with -o: ps -o pid,ppid,stat,args. The letter T in STAT means the process was stopped by a signal (SIGSTOP or SIGTSTP) and is doing nothing until it gets SIGCONT.

lsof: what files a process has open

lsof (list open files) answers the question "what does this process hold open". In Linux, an open file can mean a regular file, a socket, a device, or a pipe.

By PID:

$ lsof -p 812
COMMAND PID USER   FD   TYPE DEVICE  SIZE/OFF   NODE NAME
nginx   812  www  cwd    DIR    8,1      4096      2 /
nginx   812  www    6u  IPv4  18432       0t0    TCP *:http (LISTEN)
nginx   812  www    9u   REG    8,1     14210 262145 /var/log/nginx/access.log

The FD column is the descriptor number and mode: 6u means descriptor 6 is open for reading and writing, 3w write-only, 3r read-only. cwd is the process's current directory.

The reverse question: who's holding a port.

lsof -i :80       # processes listening on or using port 80

This is the first move when a service fails to start with "address already in use" (in Python, that's OSError: [Errno 98] Address in use): lsof -i :<port> immediately names the PID of whoever already holds the port.

By default, lsof swaps port numbers for names from /etc/services, so port 8080 shows up as *:http-alt. The flags -nP turn off host and port names, and -sTCP:LISTEN keeps only listening sockets. The -t flag prints just the PIDs, handy for scripts:

lsof -nP -iTCP:8080 -sTCP:LISTEN

COMMAND PID USER FD   TYPE DEVICE SIZE/OFF NODE NAME
python3  10 root 3u  IPv4 818881      0t0  TCP *:8080 (LISTEN)

lsof -t -i :8080

10

The same question gets broader sometimes: who holds not a single file, but an entire mount point. This is the classic umount: target is busy error, when a partition won't unmount because something inside it is still working. lsof +f -- /mnt/data lists processes with open files on that mount point, fuser -m /mnt/data gives a short list of PIDs.

$ lsof +f -- /mnt/data
COMMAND  PID USER FD   TYPE DEVICE SIZE/OFF NODE NAME
python  5120  app  3w   REG   8,17    20480  132 /mnt/data/import.csv

$ fuser -m /mnt/data
/mnt/data:            5120

Both commands work on a mount point. For an ordinary directory, lsof +f -- answers not a file system, and fuser -m lists everyone holding anything on the whole filesystem that directory lives on (often half the system). For a directory, use lsof +D /path: it walks the directory recursively.

What to do next depends on the situation: stop the service, wait for writes to finish, or signal the process. The mount point unmounts once the last holder lets go.

strace: what system calls a process makes

A system call (syscall) is a process asking the kernel for something: open a file, read a socket, allocate memory. strace intercepts these calls and prints them in real time.

Attach to an already running process by PID:

$ strace -p 812
read(6, "GET / HTTP/1.1\r\n"..., 8192) = 79
openat(AT_FDCWD, "/var/www/index.html", O_RDONLY) = 11
write(9, "192.168.1.10 - - [11/Jul...]", 64) = 64

Or start a command under strace from the beginning. The -f flag follows child processes too, -e trace=openat keeps only file opens:

strace -f -e trace=openat cat /etc/app/config.yml

strace is most useful in two situations. The process is stuck: you can see exactly which call it stopped on (say, a read from a socket or pipe that nobody is writing to). Or the process can't find a file, and a line like this shows up:

openat(AT_FDCWD, "/etc/app/config.yml", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)

ENOENT says plainly: that path doesn't exist. From there, figure out why the process expects it right there. The flags in parentheses (O_LARGEFILE and others) depend on the program and libc; they don't change what the line means.

strace -p needs permission to trace someone else's process. On Ubuntu, the Yama module is enabled by default with kernel.yama.ptrace_scope=1: a regular user can attach only to their own children, and only root can attach to anyone else. In a Docker container without the SYS_PTRACE capability, the attach fails like this:

strace: attach: ptrace(PTRACE_SEIZE, 12): Operation not permitted

Starting a command under strace from scratch (strace cmd) works without these permissions: the process is born traceable.

Here's a second trap. strace isn't free: it stops the process on every syscall, and a busy process under it slows down noticeably, sometimes by a lot. On a production service with real traffic, you attach it briefly and deliberately, rather than leaving it running.

strace and /proc exist only on Linux, not on macOS. In this course, everything here runs inside a container.

/proc: the same data straight from the kernel

The directory /proc/<PID>/ is the kernel's interface to a process's state, exposed as ordinary files. ps and lsof read much of this themselves.

cat /proc/812/status              # PID, PPid, State, memory, threads (Threads:)
tr '\0' ' ' < /proc/812/cmdline   # command line, arguments separated by \0
ls -l /proc/812/fd                # open descriptors: symlinks to files and sockets
cat /proc/812/wchan               # kernel function the process is sleeping in

ls -l /proc/812/fd shows the same thing as lsof -p 812, just without the wrapper:

lrwx------ 1 www www 64 Jul 11 09:01 6 -> socket:[18432]
l-wx------ 1 www www 64 Jul 11 09:01 9 -> /var/log/nginx/access.log

If a process is in state S and doing nothing, wchan will tell you what it's waiting for. For example, pipe_read means the process is reading from a pipe or FIFO and waiting for data. Who holds the other end will show up via lsof <path to the FIFO>: the reader has r in the FD column, the writer has w.

Signals: a few things worth remembering

Observing a process, you often end up controlling it with signals too. A couple of facts that are easy to trip over.

Exactly two signals can't be caught, blocked, or ignored: SIGKILL and SIGSTOP. The process is free to handle all the others however it likes.

A signal has a default action if no handler is set. There are five: Term (terminate), Ign (ignore), Core (terminate with a core dump), Stop (stop), Cont (continue). For example, SIGCHLD defaults to Ign: a process isn't required to react when a child exits. A process stopped by SIGSTOP resumes after SIGCONT from where it left off, not from the start.

About zombies from the previous lesson: a zombie leaves behind an entry in the process table with a PID, an exit code, and resource stats, while the process's memory has already been freed. When a parent dies before its child, the orphan gets adopted by init (PID 1), which calls wait() for it, so permanent zombies don't happen under a normal init. Since kernel 3.4, this reaper role can be handed off in advance to a designated process via subreaper, not only to PID 1.

Common mistakes

• You look for a thread in ps aux and don't find it. Threads show up only with -L or in /proc/<PID>/task/.
• You see port http-alt in lsof and don't recognize it as 8080. Add -nP.
• You kill the first process you find on a port without checking what it is. First tr '\0' ' ' < /proc/<PID>/cmdline, then decide.
• strace -p answers Operation not permitted, and it looks like strace is broken. You're missing ptrace permissions: root, sudo, or SYS_PTRACE in a container.
• You run strace without -f, and a child process is the one opening the file (a wrapper through sh, exec, fork). The line you need never shows up in the output.
• You restart a hung process instead of finding out what it's waiting for. The cause stays, and a minute later it's hung again.

What you'll need for the tasks

• docker exec -it <container> bash: get into the container. It already has ps (procps), lsof and strace.
• ps aux, ps -o pid,ppid,stat,args: the list of processes and their state.
• lsof -nP -iTCP:<port> -sTCP:LISTEN, lsof -p <PID>, lsof <file>: who holds a port, what a process holds, who holds a file.
• strace -f -e trace=openat <command>: which files a command tries to open; strace -p <PID>: what a live process is doing.
• /proc/<PID>/cmdline, status, fd/, wchan: the same data, straight from the source.
• kill -<SIGNAL> <PID>: send a process a signal (TERM, STOP, CONT).