Signals in depth: what you can catch, what you can't

You type kill, and the process keeps running. Or you hit kill -9, and the service loses half its data. Both cases come down to how signals work. In this lesson we'll look at which signals a process can catch, which it can't, and how to check that on a live process through /proc.

What a signal is

A signal is an asynchronous notification that the kernel or another process delivers to a process. Asynchronous means it arrives at an arbitrary moment, not when the process itself is waiting for it. The process might be computing, sleeping or handling a request at the time, and the signal interrupts it mid-way.

You send a signal with the kill command. The name is misleading: kill means "send a signal", not necessarily terminate.

kill -TERM 1234    # send SIGTERM to process 1234
kill -HUP 1234     # send SIGHUP
kill -s HUP 1234   # the same thing, the POSIX form

kill -l shows the full list of names and numbers. bash output:

 1) SIGHUP	 2) SIGINT	 3) SIGQUIT	 4) SIGILL	 5) SIGTRAP
 6) SIGABRT	 7) SIGBUS	 8) SIGFPE	 9) SIGKILL	10) SIGUSR1
11) SIGSEGV	12) SIGUSR2	13) SIGPIPE	14) SIGALRM	15) SIGTERM
...

The numbers in this lesson are for Linux on x86 and ARM. On some other architectures (MIPS, SPARC, Alpha) a few numbers differ, so in scripts it's safer to use the signal name rather than its number.

Five default actions

Every signal has a default action: what happens if the process hasn't set its own handler. There are five such actions in total.

Action	What it does
Term	terminate the process
Ign	ignore the signal
Core	terminate and write a core dump
Stop	suspend execution
Cont	resume a suspended process

For example, SIGSEGV (access to memory that isn't yours) has a default action of Core: the process crashes and leaves a dump for debugging. SIGCHLD (a child exited) has a default action of Ign, so by default a parent does nothing about this signal.

A custom handler for almost any signal

A process can assign its own handler function to almost any signal, and thereby override the default action. This is what graceful shutdown rests on: a daemon catches SIGTERM, flushes its data and closes connections, instead of dying instantly.

Besides a handler, a process has two more options. It can ignore a signal: the kernel will simply drop it. Or it can block it: the signal waits in a queue until the block is lifted.

In a shell script, the built-in trap command sets a handler:

trap 'echo "got HUP, reloading config"' HUP   # custom handler
trap '' QUIT                                   # ignore
trap - HUP                                     # restore the default action

Two signals can't be overridden: SIGKILL (9) and SIGSTOP (19). These are the only two a process cannot catch, block or ignore. The kernel handles them itself, bypassing the process's code.

kill -9 1234     # SIGKILL: the process never sees it, the kernel forces it off
kill -STOP 1234  # SIGSTOP: a freeze, also with no right to refuse

The shell itself might not object, though. Both bash 5.2 and BusyBox sh silently accept trap '...' KILL, but the kernel doesn't install that handler. In the masks from the next section, the SIGKILL bit stays off regardless.

How to see what a process catches

The kernel keeps each process's signal settings and shows them in /proc/<PID>/status. You need three lines:

• SigBlk — blocked signals;
• SigIgn — ignored ones;
• SigCgt — caught ones, meaning they have a handler.

grep -E 'Sig(Blk|Ign|Cgt)' /proc/1234/status

SigBlk:	0000000000000000
SigIgn:	0000000000000004
SigCgt:	0000000000010003

Each line is a bitmask in hexadecimal. Signal number N corresponds to bit N−1, counting from zero on the right. This is what a shell script with the two trap calls from the example above looks like when run under BusyBox sh. Let's break it down:

• SigIgn = 0x4 = 100 in binary. Bit 2 is set, that's signal 3, SIGQUIT. The process ignores it.
• SigCgt = 0x10003. Bits 0, 1 and 16 are set: signals 1 (SIGHUP), 2 (SIGINT) and 17 (SIGCHLD).

Our trap caught SIGHUP. But the shell (both BusyBox sh and bash) catches SIGINT and SIGCHLD on its own, with no trap involved: that's how it tracks its children and Ctrl+C. So in any shell script these two bits in SigCgt are almost always set, and they don't mean the script has a handler of its own.

Checking individual bits is easiest with shell arithmetic: echo $(( 0x10003 >> 16 & 1 )) prints 1, because bit 16 is set. kill -l 17 converts a number to a name, it prints CHLD.

The masks show what the process will react to without looking at its code. That's useful when there's no source, or it's too big to read.

Common signals

SIGTERM (15) — a polite request to exit. This is the default signal for kill when no number is given. The process is free to handle it and clean up after itself.

SIGKILL (9) — a hard termination with no chance to handle it. Any data the process hadn't written yet is lost.

SIGHUP (1) — historically "the terminal hung up". By convention, for daemons it means "reload your config", so kill -HUP is often used instead of restarting a service. nginx and sshd do this, for example. But if the process has no handler, SIGHUP's default action is Term, and the process exits.

SIGINT (2) arrives on Ctrl+C, SIGTSTP (20) arrives on Ctrl+Z. Both are generated by the terminal, and both can be caught by the process. Don't confuse SIGTSTP with SIGSTOP (19): the first is polite and can be overridden, the second can't be refused.

$ sleep 300
^C                 # the terminal sends SIGINT (2), sleep exits
$ sleep 300
^Z                 # the terminal sends SIGTSTP (20), sleep stops
[1]+  Stopped                 sleep 300

PID 1 in a container

The process with PID 1 has special rules. The kernel doesn't deliver a signal to it if it has no handler for that signal and the default action is to terminate. Exception: signals from the parent namespace, where SIGKILL and SIGSTOP still work.

In a container your application is often PID 1 itself. If it doesn't set a handler for SIGTERM, docker stop sends SIGTERM, nothing happens, and after 10 seconds Docker finishes the process off with SIGKILL:

time docker stop sleepy

sleepy

real	0m10.2s

Details on init in a container are coming in the Docker module.

Common mistakes

"kill always kills." By default kill sends SIGTERM, and the process can catch it and, say, ignore it. If the process doesn't react to SIGTERM, it keeps running. Check SigIgn and SigCgt before reaching for -9.

"kill -9 terminates cleanly." The opposite: SIGKILL tears the process apart wherever it happens to be. A half-written file, a lock that never got released, a database connection cut mid-flight. The right order: SIGTERM first, give it time for a graceful shutdown, and only reach for SIGKILL on a process that's truly stuck.

Signal to the wrong process. A script starts sleep, and ps shows both processes. Send SIGHUP to the sleep instead of the script, and sleep dies while the script notices nothing. Look at the command in ps, not the first PID you see.

kill -HUP to a process with no handler. For a daemon that can't reload its config, SIGHUP means termination. Before sending it in production, check the daemon's documentation, or bit 0 in SigCgt.

Orphans and zombies

When a parent dies before its child, the child becomes an orphan and gets adopted by init (PID 1). Init receives SIGCHLD when an adopted child exits and collects its exit code with wait(), so they don't hang around for long. Since kernel 3.4, the reaper role can be delegated from PID 1 to an intermediate subreaper process via prctl(PR_SET_CHILD_SUBREAPER).

A zombie is a process that has exited but whose exit code its parent hasn't yet collected with wait(). It holds an entry in the process table (a PID number) but no memory. It costs almost no resources; the danger is only in numbers: if the parent never calls wait(), zombies pile up and can exhaust the PID table.

ps aux | awk '$8 ~ /Z/'

user  4821  0.0  0.0   0    0 ?  Z   10:22  0:00 [worker] <defunct>

We'll cover orphans and zombies in detail in the next lesson.

What you'll need for the tasks

• ps inside the container, to find the PID of the process you need by its command.
• kill -<SIGNAL> <PID> and kill -l, to convert a number to a name.
• grep -E 'Sig(Ign|Cgt)' /proc/<PID>/status and converting the mask to signal numbers: signal N is bit N−1.
• docker exec <container> <command>, to run a command inside the lab container.