Zombies and orphans: reaping

ps shows lines with status Z, kill -9 on them changes nothing, and in a container they pile up into the hundreds over time. To understand where they come from and how to get rid of them, you need to know what happens to a process after it finishes and who is responsible for cleaning up after it. In this lesson we'll cover zombies, orphans, and why in a container this chore falls on your application.

What happens when a process exits

A process doesn't disappear from the system the moment its code finishes running. First it moves into a terminated state, but a record of it stays in the kernel's process table. That record holds the exit code: the number the process handed over through exit().

The parent has to collect that code. It does this with the wait() system call (or waitpid()). Until the parent collects the code, the record stays in the table.

You see the exit code every day: the shell calls wait() after each command and puts the result in $?.

echo $?      # exit code of the last finished command

0

Reaping, and where a zombie comes from

Reaping is when the parent calls wait() and collects the child's exit code. After reaping, the kernel drops the record from the process table, and the PID is freed.

A zombie is a terminated process whose exit code nobody has collected yet. In ps it shows state Z.

Here's what a zombie looks like in ps from the procps package (Ubuntu, Debian, Fedora):

ps -eo pid,ppid,stat,args

    PID    PPID STAT COMMAND
      9       1 S    sleep 3
     11       9 Z    [sleep] <defunct>

Here process 11 is already dead, but its parent 9 isn't calling wait(). The record stays, PID 11 is held. procps puts the <defunct> tag in the full-command column (args, cmd, the output of ps aux); the comm column just shows sleep. BusyBox, which ships in alpine images, writes a zombie's name in square brackets with no tag at all:

PID   PPID  STAT COMMAND
    8     1 Z    [sh]

The most reliable signal is the Z letter in the STAT column.

A zombie holds a record in the process table and a PID number. Memory, files and the stack were already freed by the kernel when it terminated. All that's left is a row in the table.

The kernel notifies the parent of a child's death with the SIGCHLD signal. Its default action is Ign: nothing happens, and the parent has to either catch SIGCHLD and call wait(), or poll its children periodically with waitpid() and the WNOHANG flag.

There's a third way: explicitly set the handler for SIGCHLD to SIG_IGN. Formally this is also "ignoring", but in this case the kernel reaps terminated children itself, and no zombies appear. The cost: exit codes are lost, wait() will never get them.

Why zombies are dangerous

One zombie is harmless. The problem starts when a parent has a bug and never calls wait(). Then every child that exits leaves behind a record, and they pile up.

PID numbers are a finite resource. The system-wide ceiling is visible here:

cat /proc/sys/kernel/pid_max

4194304

The kernel defaults to 32768, and distributions running systemd raise it to 4194304 on 64-bit systems. In a container the ceiling is usually lower: Docker and Kubernetes can limit the number of processes through a cgroup (docker run --pids-limit). A container's limit is visible in /sys/fs/cgroup/pids.max, and a value of max means "no limit".

When zombies fill the table to the limit, the system can't hand out a PID to a new process, and fork() returns the error EAGAIN. At the scale of a whole machine this rarely happens, but in a container with a --pids-limit of a few hundred, a buggy process will exhaust the limit within minutes.

Orphaned processes

An orphaned process (orphan) is a live process whose parent terminated before it did. Such a process doesn't go unsupervised: the kernel assigns it a new parent.

  PID  PPID COMMAND
 5310     1 sleep       # was a child of the shell, the shell closed, PPID became 1

Here sleep outlived its original parent, and its PPID became 1. The process with PID 1 (that's init, usually systemd) adopts orphans. When such an adopted child exits, init gets SIGCHLD, calls wait(), and removes the record. That's why an orphan under init doesn't turn into a permanent zombie.

If a parent died while its children were already zombies, they also move to init, and it removes them right away. So a permanent zombie is fixed by stopping its parent.

Subreaper

PID 1 isn't the only one that can adopt orphans. Since kernel 3.4, a process can declare itself a subreaper with a prctl() call:

prctl(PR_SET_CHILD_SUBREAPER, 1);

After that, the kernel hands orphans from its subtree to it instead of PID 1. More precisely, the kernel walks up the dead parent's ancestors and takes the nearest subreaper. If there's none, the orphan goes to init.

This is how process managers and init wrappers work. For example, on a Linux desktop a background job started from a terminal often ends up with a PPID of not 1 but systemd --user after the terminal closes: the user's systemd manager declares itself a subreaper. So a PPID of 1 for an orphan isn't guaranteed; check which process actually sits behind that number.

PID 1 in a container

In a container, the first process (PID 1) isn't init, it's your application. The role of "orphan cleaner" falls to it. If the application doesn't call wait() for all its children, adopted orphans turn into zombies after they die and pile up inside the container.

Orphans show up in a container more often than you'd think: any shell script with a & that exits before its background job does leaves behind an orphan. Here's a container where PID 1 is a Python program that runs such scripts and waits only for them, not for the ones they orphan:

PID   PPID  STAT COMMAND
    1     0 R    python3 app.py
    8     1 Z    [sleep]
   10     1 Z    [sleep]
   12     1 Z    [sleep]

There are two ways out. First: have the application collect all terminated children itself, by catching SIGCHLD or calling waitpid(-1, WNOHANG) in a loop. Second: put a small init process in the PID 1 slot, one whose only job is reaping and signal forwarding (tini, or the docker run --init flag, which runs Docker's built-in docker-init).

PID   PPID  STAT COMMAND
    1     0 S    /sbin/docker-init -- python3 app.py
    7     1 R    python3 app.py

With --init, the application gets PID 7, and docker-init adopts orphans and reaps them right away.

Common mistakes

The myth "a zombie eats memory" is wrong. The process's memory was freed when it terminated; all that's held is a PID and a table record.

The myth "kill -9 will remove a zombie" is also wrong. A zombie is already dead, a signal won't do anything to it:

kill -9 11       # zombie process
ps -o pid,stat -p 11

    PID STAT
     11 Z

Only the parent's wait() removes a zombie. If the parent is stuck and not calling wait(), stopping or restarting that parent helps: its children get adopted by init and reaped. In a container, where the parent is PID 1, that means restarting the container, and zombies will start piling up again until the root cause is fixed.

One more mistake: looking for zombies only among the application's direct children. In a container, zombies are often someone else's: orphans that PID 1 adopted. Look at the PPID column, it shows who should have called wait().

The last one: removing zombies with SIG_IGN for SIGCHLD and forgetting the cost. The zombies will go away, but so will the children's exit codes: code that checks whether a command succeeded will start seeing success where there was actually a failure.

What you'll need for the tasks

• ps -o pid,ppid,stat,args inside a container: state Z in the STAT column and the parent in the PPID column.
• docker compose ... exec <service> <command> and docker exec <container> <command>, to run a command inside the lab environment's container.
• How to put an init process in the PID 1 slot (--init on docker run, the matching key in compose).
• kill <PID> and understanding where orphans go once their parent dies.