Process lifecycle: fork, exec, exit

How a process is born: fork, exec, exit, the process tree and exit codes.

Why this matters

You type ls, and a fraction of a second later the shell is waiting for a command again. In that time the kernel managed to create a process, load a different program into it, wait for it to finish, and hand the shell an exit code. Almost everything in operations rests on this chain: deploy scripts decide from the exit code whether to go on or not, CI colors a step red based on the same thing, and docker ps shows Exited (137), a number you need to be able to read.

This lesson continues "Processes and signals" from the basics track. That one covered how to find a process and stop it. Here we'll look at where a process comes from and how it ends.

How a process is born

A process isn't built from scratch. The kernel creates a new process by copying one that already exists. The system call fork makes a duplicate of the calling process: the copy gets a new PID, but has a copy of the parent's memory, the same open files and the same current directory. After fork there are two processes, and both keep running from the line right after the call.

From that point their memory is their own. If the child changes a variable, the parent won't see the change.

Telling the parent from the child is simple: fork returns a different value. The parent gets the child's PID, the child gets zero.

pid = fork();          # both parent and child are here
# in the parent pid = 4210 (the child's PID)
# in the child  pid = 0

exec replaces the image

A duplicate of the parent is rarely needed on its own. Usually the child wants to run a different program. For this it calls exec. This is a family of functions (execl, execvp and others), all backed by one system call, execve. It replaces the current process's image with a new program. The PID doesn't change, the contents do: code, data, stack.

# inside the child after fork
exec("/bin/ls", ...);  # the process stops being a copy of the shell
                       # and becomes ls, same PID

This is where the fork+exec model comes from. The shell forks, and the child calls exec with the command you asked for. The parent shell stays alive and waits for the child to finish.

$ ls        # bash forks, the child does exec("ls"), bash waits
file1  file2
$           # the child finished, control returned to bash

exec can also be called directly from the shell. The command exec program doesn't create a child: the shell itself turns into that program and won't come back after it ends. Container entrypoint scripts often end this way (exec python app.py), so the application takes over the script's PID and no extra wrapper process is left behind.

PID, PPID and the tree

A process has two numbers that set its place in the system. PID is the process's own unique number. PPID is its parent's PID. These links arrange processes into a tree: the terminal spawned the shell, the shell spawned the command.

$ echo $$      # PID of the current shell
3187
$ bash         # start a nested shell
$ echo $$      # it has its own PID
4055
$ echo $PPID   # and its parent is the previous shell
3187

The whole tree is shown by pstree from the psmisc package. PID 1 (usually systemd) sits at the root, and everything else branches off it.

$ pstree -p
systemd(1)---sshd(880)---sshd(3140)---bash(3187)---pstree(4102)

On minimal systems and in containers pstree is often missing. Then ps shows the PID and PPID pairs:

ps -o pid,ppid,args

From the PPID column you reconstruct the tree by hand: you look for the line whose PID equals the PPID you need, and climb up to PID 1.

exit and exit codes

A process ends by calling exit with an integer, called the exit code. Zero means success, any nonzero value signals an error. The parent reads this code and decides what to do next.

In the shell, the last command's code lives in the variable $?.

$ ls /etc/hostname
/etc/hostname
$ echo $?      # success
0
$ ls /no-such-file
ls: cannot access '/no-such-file': No such file or directory
$ echo $?      # error
2

The specific nonzero number is up to the program itself. GNU ls on Linux responds to a nonexistent path with 2, while ls on macOS responds with 1. That's why scripts check "zero or nonzero" rather than a specific digit, unless the program documents it.

The code occupies one byte, from 0 to 255. exit 256 gives 0, exit 300 gives 44: only the remainder of division by 256 is left.

If a process is killed by a signal instead of calling exit itself, the shell shows the code 128 + signal number. SIGKILL (9) gives 137, SIGTERM (15) gives 143. You'll see the same numbers in docker ps for a container whose main process was killed, for example Exited (137).

$ sh -c 'kill -9 $$'; echo $?
137

One more detail that breaks scripts: $? holds the code of the last command. A function or a script without an explicit exit returns the code of its last line. If that last line is echo, the code will be 0, even if the command before it failed.

You need to grab the code right away, before the next command overwrites it. There are two common ways. First: save it to a variable and exit with it explicitly. Second: set -e at the top of the script, then the shell itself ends the script on the first failed command with its code.

cp app.tar.gz /backup/
rc=$?
echo "copy finished"
exit $rc

Termination signals and orphaned processes

A process can also be ended with a signal. There are two signals a process cannot catch, block or ignore: SIGKILL and SIGSTOP. This is the only such pair; for every other signal you can set a handler.

Every signal has a default action, five in total: terminate the process (Term), ignore it (Ign), terminate with a memory dump for debugging (Core), stop it (Stop), continue a stopped one (Cont). SIGCHLD, which the kernel uses to tell the parent a child has died, defaults to Ign.

When a child has ended but the parent hasn't yet read its code through wait(), it remains a zombie. It holds an entry in the process table, that is, a reserved PID number, but no memory. As soon as the parent calls wait(), the entry is removed.

$ ps aux | grep defunct
deploy  4310  0.0  0.0  0  0 ?  Z  10:20  0:00 [worker] <defunct>

If the parent dies before the child, the child becomes orphaned. Such an orphan is adopted by the process with PID 1 (usually init/systemd), which also calls wait(), collecting the exit code. Since kernel 3.4 the role of adopter can also be assigned to an ordinary process via a subreaper, so orphans go to it instead of PID 1. Zombies and orphans will be covered in detail in lesson 4 of this track.

Common mistakes

• "A process is created from scratch." No, fork copies the parent with all its environment: variables, open files, current directory.
• "exec spawns a new process." No, exec replaces the current process's image, the PID stays the same. Only fork gives a new process.
• "The script failed, so CI will see it." Only if the failed code made it to the end. A function that does echo after the command, and a script that ends with echo "done", will both return 0.
• "Code 137 means an error in the program." No, 137 = 128 + 9: the process was killed by SIGKILL, often due to running out of memory or a stop timeout.
• "pstree is everywhere." In containers and minimal images it's usually missing; use ps -o pid,ppid,args instead.

What you'll need for the tasks

• echo $$, echo $PPID and ps -o pid,ppid,args: find a process and its parent.
• The difference between fork and exec in practice: after exec the process has the same PID, but a different program in the args column.
• $? and the "code of the last command" rule: how a function or script loses the code of a failed step.
• exit N and set -e in shell scripts.
• For the first lab, Docker 20.10 or newer; for the second, sh is enough.

What to remember

fork makes a copy of the parent: a new PID, a copy of the memory, the same open files.

exec replaces the image with another program, the PID doesn't change. This is where the fork+exec model comes from: the shell forks, the child does exec.

PID is the process's number, PPID is the parent's number. pstree draws the tree from PID 1; without it, ps -o pid,ppid,args is enough.

exit gives an exit code from 0 to 255: 0 means success, nonzero means an error. Check $?. A code above 128 usually means the process was killed by a signal.

SIGKILL and SIGSTOP can't be caught or ignored. A zombie holds a PID number, and PID 1 adopts an orphan.

The next lesson covers process states and the scheduler: what R, S, D mean and how the kernel hands out the CPU to them.