Processes and signals

PID, ps and top, how SIGTERM differs from SIGKILL, and how to stop a process correctly.

Why this matters

Almost any work on a server eventually runs into processes. A service hangs, and you need to stop it. A restart hangs for a minute and a half, and you need to figure out why. Lines with the letter Z show up in ps, and a colleague asks if that's dangerous. All of this comes down to three things from this lesson: find the process, understand its state, and send it the right signal.

What a process is

A process is a running program: its code, the memory allocated to it, its open files, and its execution context. You can run the same binary ten times and get ten different processes.

Every process has a number, its PID (process id). PID 1 is special: it's the first process the kernel starts when the system boots (usually systemd). Every other process branches off from it in a tree.

Processes don't appear out of nowhere: one process (the parent) spawns another (the child). A child has a PPID, the PID of its parent. That's how the tree forms: a terminal starts your shell, and the shell starts the scripts you run in it.

A container has its own numbering. PID 1 there isn't systemd, it's whatever the container started: your application, or a small init like docker-init (added by the docker run --init flag). This matters for the tasks.

What to use to look at processes

ps aux                 # a snapshot of all processes at once
ps aux | grep nginx    # filter by name
top                    # a live monitor, updates every 3 seconds by default
htop                   # the same, but nicer (needs installing separately)
pgrep -a nginx         # PID by name, plus the command line

Here's what ps aux output looks like:

USER   PID %CPU %MEM    VSZ   RSS STAT START  TIME COMMAND
root     1  0.0  0.1 168240 11500 Ss   09:00  0:02 /sbin/init
www     812  1.3  2.4 720160 98400 Sl   09:01  0:21 nginx: worker
deploy 1455  0.0  0.0  12100  3200 R+   10:14  0:00 ps aux

Useful columns: PID (the process number), %CPU and %MEM (load), RSS (memory actually used, in KB), STAT (state), COMMAND (what's running). In top, load average and used memory show up at the top.

The first letter in STAT is the state:

• R — running or ready to run;
• S — sleeping, waiting for an event (the normal state for most processes);
• D — uninterruptible sleep, usually waiting on disk (a worrying sign if it lasts long);
• Z — zombie (see below);
• T — stopped (for example, by Ctrl+Z).

The following letters add detail: s — session leader, l — multi-threaded, + — in the terminal's foreground group. That's why the example shows Ss, Sl, and R+. The full list is in man ps, section PROCESS STATE CODES.

You can see the parent if you ask for the PPID column explicitly:

ps -o pid,ppid,stat,args

ps works this way even on stripped-down systems like the busybox image, where ps aux prints only PID, USER, TIME, and COMMAND.

Signals

A signal is an asynchronous notification that the kernel (or another process) delivers to a process. You send one with the kill <PID> command. Despite the name, kill means "send a signal", not necessarily kill.

The three most important ones:

Signal	Number	Meaning
SIGTERM	15	"finish up cleanly". kill's default signal
SIGKILL	9	"die right now". The process can't catch it
SIGHUP	1	"reread your config" (by convention for daemons) or "the terminal closed"

Ctrl+C in a terminal sends another signal, SIGINT (2).

kill 1234           # send SIGTERM to process 1234
kill -9 1234        # send SIGKILL
kill -HUP 1234      # send SIGHUP (reread the config)
pkill nginx         # by name, not by PID

Why kill -9 is the last resort

A process can catch and handle SIGTERM: flush data to disk, close database connections, finish requests cleanly, delete temp files. This is called a graceful shutdown.

SIGKILL (-9) can't be caught: the kernel terminates the process immediately, and not another line of its code runs. The consequences:

• whatever the program held in memory and hadn't written yet is lost;
• the kernel closes files and sockets, but the application never says goodbye: clients see a dropped connection, an unfinished database transaction rolls back;
• broken state can remain: a half-written file, a lock file nobody released.

The right order is: send SIGTERM first, then wait (in practice 5-30 seconds, depending on the service), and only send SIGKILL if the process is well and truly stuck. That's how docker stop and systemctl stop both work, just with different timeouts.

docker stop waits 10 seconds by default, changed with the -t flag. For systemd, the countdown comes from TimeoutStopSec= in the unit, and the default is DefaultTimeoutStopSec from /etc/systemd/system.conf, which is 90 seconds. That's where the classic complaint "the service restart hangs for a minute and a half" comes from: the service doesn't respond to SIGTERM, and systemd dutifully waits out its timeout before sending SIGKILL.

Zombie processes

When a process exits, it doesn't disappear instantly. Its entry stays in the process table until the parent collects its exit code with a wait() call. A process that has finished but hasn't been cleared this way is called a zombie (Z in ps, and procps appends <defunct> to its name).

A zombie uses no CPU or application memory: it's already dead. All it's holding onto is a slot in the table. But if the parent is broken and never calls wait(), zombies pile up and can exhaust the PID limit.

A signal can't clear a zombie. Even kill -9 does nothing to it: there's no one left to kill. The fix is restarting the parent. When a parent dies, its children are adopted by PID 1 (or the nearest process set up as a "subreaper", if one is configured). The new parent gets SIGCHLD, calls wait(), and the zombies disappear from the table.

This matters in containers: PID 1 there is your application, and it usually doesn't collect other processes' children. That's why containers whose processes spawn children run a small init: docker run --init or tini inside the image.

Background processes

./long-script.sh &        # run in the background, return control to the shell
jobs                      # list the current shell's background jobs
fg %1                     # bring job 1 to the foreground

The problem: a background process stays a child of the shell. Close the terminal, and the shell gets SIGHUP and passes it on to its jobs. To make a process survive the session closing:

nohup ./long-script.sh &       # ignore SIGHUP, log to nohup.out
disown %1                      # detach an already-running job from the shell

disown is a bash and zsh builtin. dash (that's /bin/sh on Debian and Ubuntu) doesn't have it, so there you use nohup and setsid instead.

In production, long-running services aren't started this way. They're set up as a systemd unit that watches the process and restarts it if it crashes. nohup and & are fine for one-off tasks.

Common mistakes

• Reaching for kill -9 right away. The process doesn't get to flush its data, leaving lock files and half-written files behind. Try plain kill first, wait, and only then -9.
• kill -9 on a zombie. Nothing happens: the zombie has already exited. Look at the PPID column and deal with the parent.
• Stopping the whole container instead of one process. docker stop sends a signal to PID 1, and every process in the container goes down with it. If one process is the problem, send the signal to that one through docker exec <container> kill <PID>.
• Sending a signal to PID 1 inside a container. If PID 1 exits, the whole container stops along with every process in it.
• Remembering a PID and reusing it next time. Numbers are reassigned on every run. Check ps again before each kill.

What you'll need for the tasks

Both labs run in a container on the busybox:1.36.1 image, so they work the same way on Linux, macOS, and WSL2. You'll need:

• docker exec <container> ps -o pid,ppid,stat,args — the process list with parents and state;
• docker exec <container> kill <PID> and docker exec <container> kill -9 <PID>;
• docker exec <container> cat <file> — look at a file inside the container;
• the sections "Why kill -9 is the last resort" and "Zombie processes".

What to remember

A process is a running program with its own PID. The tree grows from PID 1.

To look at processes: ps aux, top/htop, pgrep. The parent is in the PPID column.

SIGTERM (15) asks a process to finish cleanly, SIGKILL (9) kills it with no chance to catch it. Try TERM first, KILL only as a last resort.

A zombie is a finished process the parent never cleared. A signal doesn't fix it; restarting the parent does.

A long-running service needs a systemd unit, not nohup &.

The next lesson covers file permissions: who can do what with a file.

Once this sinks in, continue in the "Linux: processes in depth" track: fork and exec, process states, zombies, and strace.