Graceful shutdown: SIGTERM and connection draining

Why this matters

Every rollout (rolling, blue-green) stops the old replica of the application. The orchestrator doesn't kill the process right away: it sends SIGTERM and waits. If the application knows how to react, it has time to finish the requests it's already handling. If not, the orchestrator sends SIGKILL after a timeout, and every open connection breaks along with whatever request it was carrying.

Graceful shutdown means shutting down cleanly: stop taking new work, finish what's already in progress, release resources.

Signals

SIGTERM (15) asks a process to terminate, and it can be caught and handled. SIGKILL (9) can't be caught; the process dies instantly. The orchestrator and docker stop both send SIGTERM first, wait, and only then send SIGKILL. How long they wait depends on the system: docker stop waits 10 seconds by default for Linux containers, Kubernetes waits 30 seconds.

The exit code hints at how the process died. If a signal killed it, the code is 128 + the signal number: 137 means SIGKILL, 143 means SIGTERM with no handler. A clean exit gives 0.

docker ps -a --filter name=shop --format '{{.Names}}  {{.Status}}'

shop  Exited (137) 3 seconds ago

The application's job: catch SIGTERM and wind down correctly within the time it's given.

The order of a clean shutdown

When SIGTERM arrives, the steps are:

1. Stop accepting new work. Close the listening socket so new requests stop arriving.
2. Finish what's in progress. Let the current requests complete (this is draining).
3. Close resources. Database connections, files, queue producers.
4. Exit with code 0.

Roughly, in pseudocode:

on SIGTERM:
    server.stop_accepting()      # step 1
    server.wait_active_requests(timeout=25s)  # step 2
    db.close(); queue.close()    # step 3
    exit(0)                      # step 4

Where the 502s during a "clean" shutdown come from

The first step sounds harmless, but it's exactly where bugs usually show up. The logic of "close the socket, and the load balancer will pull the replica out of rotation on its own" doesn't work, because the order of events runs the other way.

In Kubernetes, deleting a Pod starts two processes in parallel, and nothing synchronizes them:

• kubelet starts stopping the container (the preStop hook, if there is one, then SIGTERM);
• a controller removes the Pod from the Service's EndpointSlice, and from there the change propagates to kube-proxy on every node, to Ingress controllers, and to external load balancers.

The first takes milliseconds; the second takes anywhere from hundreds of milliseconds to several seconds, depending on cluster size. That leaves a window where the application has already closed its socket, but traffic is still arriving. Every such request hits connection refused, and the client sees a 502 or 503 on an otherwise clean deploy.

The fix isn't in the code, it's a pause before shutdown. The application needs to keep serving requests for a few more seconds after the stop begins, exactly as long as it takes the endpoint-removal information to propagate (propagation delay).

The simplest fix is a preStop hook. Kubernetes runs it before SIGTERM and sends the signal only after the hook finishes. The removal from endpoints is already underway by then:

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: app
      image: example/app:1.4.2
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "10"]

Notice the levels: terminationGracePeriodSeconds is set for the whole Pod, lifecycle for a specific container. During those 10 seconds the container keeps working as usual and answers requests. By the time SIGTERM arrives, the load balancers already know the replica is gone and won't send new connections.

exec: sleep requires the sleep program to be in the image. Minimal images (distroless, scratch) don't have it. For those, Kubernetes has a built-in preStop.sleep action with a seconds field, which needs no program in the image (see Container Lifecycle Hooks).

A second option, if you'd rather not touch the manifest: the SIGTERM handler first flips readiness to "not ready," sleeps for a few seconds, and only then closes the socket. The effect is the same.

Don't forget about terminationGracePeriodSeconds: the countdown starts when the stop begins, and the preStop pause counts against it. Ten seconds of sleep plus twenty-five for draining won't fit into the default thirty, and the application gets killed halfway through.

Timeout: application side and orchestrator side

There are two related numbers here. The orchestrator waits out a grace period between SIGTERM and SIGKILL. The application has to fit its drain into that same window. If the application needs 25 seconds and the grace period is 10, the orchestrator kills it halfway through. That's why the grace period is set to the longest request, with some margin.

Where it's set:

Where	Setting
Kubernetes	terminationGracePeriodSeconds on the Pod
Docker	docker stop -t, docker run --stop-timeout
Docker Compose	stop_grace_period on the service
systemd	TimeoutStopSec in the unit

The grace period is an upper bound, not a mandatory wait. An application that finishes its requests early exits on its own, and the stop takes only as long as the drain actually lasted.

The PID 1 mistake

docker stop sends the signal only to the container's main process, PID 1. PID 1 is special: the kernel doesn't apply a signal's default action to it. If the process has no handler of its own for SIGTERM, the signal is simply ignored (more on this in the lesson on processes and signals).

This leads to a common trap. The application is launched through a start script:

#!/bin/sh
echo "starting app"
python /app/app.py

PID 1 here is the shell, and python is its child process. The shell ignores the signal, it never reaches the application, and once the grace period runs out Docker kills everything with SIGKILL. The SIGTERM handler exists in the code, but it never fires. The stop takes the full 10 seconds and ends with exit code 137.

The fix is the word exec on the script's last line: exec python /app/app.py. The shell replaces itself with the application, which keeps the same PID and becomes PID 1. Lines after exec never run.

The same story happens with the shell form in a Dockerfile. CMD myapp turns into /bin/sh -c "myapp". Whether the shell stays as PID 1 depends on which sh is in the image: dash on Debian and Ubuntu stays PID 1, while BusyBox's sh on Alpine replaces itself with the program for a single command. You shouldn't rely on that, so launch the application with the exec form (CMD ["myapp"]), and it gets signals directly from the start.

To check who PID 1 is, from inside the container:

docker exec shop ps

PID   USER     TIME  COMMAND
    1 root      0:00 sh /app/start.sh
    7 root      0:00 python /app/app.py

Common mistakes

• No SIGTERM handler in the application that's PID 1. The signal is ignored, the stop waits out the timeout, and ends in SIGKILL.
• A start script without exec. The handler exists in the code, but the shell gets the signal instead.
• The socket closes right on SIGTERM in Kubernetes. Load balancers are still sending traffic, and clients get 502s. You need a pause through preStop or inside the handler.
• The grace period is shorter than the drain. Long requests get cut off, exit code 137.
• preStop isn't accounted for in the grace period. The pause eats into the time, leaving none for draining.

What to remember

The orchestrator sends SIGTERM, waits out the grace period, and only then sends SIGKILL. During that time the application needs to stop accepting new work, finish what's in progress, close its resources, and exit with code 0.

Match the application's drain time to the orchestrator's grace period, or it gets killed halfway through. Inside the container, the application needs to be PID 1 to get signals directly: the exec form of CMD, or exec in a start script.

Don't close the listening socket right on SIGTERM in Kubernetes: the endpoint removal runs in parallel with the signal and reaches the load balancers later. A pause through preStop (or a sleep inside the handler itself) closes that window and gets rid of the 502s during a deploy.

The next lesson covers health endpoints: how the orchestrator and the load balancer know the application is alive and ready.

What you'll need for the tasks

• docker compose stop, ps -a, logs, exec <service> ps, up -d (with --force-recreate if you only changed mounted files).
• Reading exit codes: 0, 137, 143.
• exec in a shell script.
• The stop_grace_period field in compose.