Daemons and background processes

You started a long script on a server over ssh, the connection dropped, and the script died with the session. This lesson covers why that happens, how & differs from nohup, disown, and setsid, and why long-running services run under systemd.

What a daemon is

A daemon is a background process detached from a terminal, running for a long time. A web server, cron, and sshd start once when the system boots and keep running until stopped. A process like that has no controlling terminal: there's no one to press Ctrl+C, nowhere to write output.

Detachment from a terminal is easy to spot in the TTY column of ps. A regular command shows a terminal name there (pts/0), a daemon shows a question mark:

ps -o pid,tty,comm -C sshd

  PID TT       COMMAND
  742 ?        sshd

The ? means the process isn't attached to any terminal. Closing any session won't affect it.

Session, terminal, and SIGHUP

Every process belongs to a session. A shell opens a session when you log into the system, and every command started from that shell lands in the same one. A session has a controlling terminal: the emulator window or pty that sshd handed out.

When the terminal goes away (you closed the window, the ssh connection dropped), the kernel sends the session leader the SIGHUP signal ("the terminal closed"). The session leader is usually an interactive bash. On getting SIGHUP, bash forwards it to all of its jobs, running and stopped, and only then exits. The default action for SIGHUP is to terminate the process.

The session number shows up in the SID column:

ps -o pid,ppid,sid,tty,comm

Every process started from the same terminal will show the same SID and the same TTY.

An ampersand backgrounds, but doesn't detach

Starting a command with an ampersand (&) sends the process to the background and returns control to the shell right away:

./long-script.sh &

[1] 4211

jobs

[1]+  Running                 ./long-script.sh &

A background job stays in the same session, with the same terminal, and is listed in the shell's job list. On a dropped connection or a closed window, bash gets SIGHUP, forwards it to the job, and the script dies with the session. So the idea that "& survives an ssh drop" is wrong.

One subtlety: if you exit cleanly, with the exit command, bash by default doesn't send jobs SIGHUP, and the background process often survives. This depends on the huponexit option (in a login shell with shopt -s huponexit, jobs get SIGHUP on exit too) and on what the process does with output to an already-closed terminal. Don't rely on that kind of luck: a connection drops without warning.

nohup and disown

To make a process survive the session closing, there are two tools. nohup starts a command with SIGHUP ignored. If output was going to the terminal, nohup redirects it to a file nohup.out in the current folder (if it can't create one there, it writes to $HOME/nohup.out):

nohup ./long-script.sh &

[1] 4230
nohup: ignoring input and appending output to 'nohup.out'

The kernel simply doesn't deliver a signal that the process ignores. So SIGHUP from bash never reaches such a process. If output is already redirected (nohup ./long-script.sh > backup.log 2>&1 &), the nohup.out file isn't created.

disown works on a task that's already running. It removes it from the shell's job list, and on a session drop bash won't forward it SIGHUP:

./long-script.sh &
disown %1

Here %1 is the job number from jobs' output. If the script's output was going to the terminal, after the drop there's nowhere to write to: the output is lost. disown -h %1 keeps the job in the list but protects it from SIGHUP the same way.

disown is built into bash and zsh, but dash (which is /bin/sh on Debian and Ubuntu) doesn't have it. There you reach for nohup or setsid.

How a process becomes a daemon

The classic scheme from man 7 daemon goes like this. The process calls fork, the parent exits right away. The child calls setsid and opens a new session, where it has no controlling terminal. Then the process resets its working directory to /, so it doesn't keep some random mount point busy, and reassigns stdin, stdout, and stderr to /dev/null. man 7 daemon has more steps (a second fork, umask, closing extra file descriptors), but the idea is the same.

The setsid command does the main step by hand:

setsid ./worker.sh > worker.log 2>&1 < /dev/null

A process started this way immediately gets ? in the TTY column, its own SID, and doesn't depend on the shell that started it. SIGHUP from another session never reaches it. The redirection here matters: setsid itself doesn't touch output, and without > the process keeps writing to the old terminal as long as it's alive.

Production runs systemd, not nohup

Long-running services don't stay on nohup. Nobody watches a process like that or restarts it when it crashes, output piles up in a single file, and after a host reboot nothing comes back up. A service gets set up as a systemd unit: systemd starts it at boot, restarts it on a crash, and writes its logs to the journal.

systemctl status myapp

The command shows whether the service is running and whether systemd is watching it.

nohup, disown, and setsid remain useful for one-off tasks: run a long script and leave the host without waiting for it to finish. tmux and screen fit the same tasks: they keep a session alive, and you can come back to it.

More on signals and daemons

SIGKILL and SIGSTOP are the only signals that can't be caught, blocked, or ignored. A process is free to handle every other one its own way, including SIGHUP.

Every signal has a default action for when the process hasn't caught it. There are five such actions:

Action	What it does
Term	terminate the process
Ign	ignore the signal
Core	terminate and dump core
Stop	suspend the process
Cont	resume a suspended process

SIGCHLD defaults to Ign: the kernel sends it to the parent when a child exits, but the parent is free to ignore it. If the parent never calls wait(), the child stays a zombie, taking up an entry in the process table (a PID), not memory.

An orphaned process, one whose parent has died, gets adopted by init (PID 1), which calls wait() on its behalf, clearing the zombie from the table. Since kernel 3.4, the adopter role can be handed to a specially designated subreaper process (prctl(PR_SET_CHILD_SUBREAPER)). That's what the user service manager systemd --user does, for example. In a container with --init, tini plays the role of PID 1 and adopter: in this lesson's labs, orphaned tasks have PPID equal to 1.

Common mistakes

• You started a long task with & over ssh and hope it survives a dropped connection. It won't: you need nohup, disown, setsid, or tmux.
• You ran nohup without & and expect the terminal to free up. nohup only protects against SIGHUP, & is what sends the command to the background.
• You're looking for nohup's output in the wrong place. nohup.out is created in the folder you ran it from, and only if the output was going to the terminal.
• You write disown in a script under #!/bin/sh on Ubuntu. dash doesn't have that command, you'll get disown: not found.
• You're keeping a service on nohup in production. After a reboot or a crash it won't come back on its own: you need a systemd unit.

What you'll need for the tasks

• docker exec -it <container> bash: get into a container with an interactive shell.
• &, jobs, nohup, disown, setsid: ways to run a process in the background and detach it from the session.
• ps -o pid,ppid,sid,tty,comm: see a process's session and terminal; ps -ef or ps -o ... -C <name> to find a process.
• kill -HUP $$: send the current shell SIGHUP, the same as on a dropped connection ($$ is the current shell's PID).
• grep SigIgn /proc/<PID>/status: the mask of ignored signals. It's a number in hex, each bit maps to a signal: bit 0 is signal 1 (SIGHUP), bit 1 is signal 2 (SIGINT), and so on.