systemd: units and services

When a server boots, something has to start nginx, the database, your app, and keep an eye on them so they don't die. On most distributions that's systemd: the system's first process (PID 1), the root of the whole process tree.

Starting a service by hand with nohup ./app & is a bad idea. The process dies on reboot, nothing brings it back after a crash, and logs scatter across random files. systemd fixes this: you describe the service once, and the system starts it, restarts it, and collects its logs on its own.

Where to practice

systemd ships with Ubuntu, Debian, Fedora, RHEL, and almost every server distribution. macOS doesn't have it; it uses its own manager, launchd. In WSL2, systemd is controlled by the line systemd=true in the [boot] section of /etc/wsl.conf, and it's already on in Ubuntu installed through wsl --install. More on that in the WSL lesson.

A plain docker run ubuntu:24.04 has no systemd either: PID 1 there is your bash. So the labs in this lesson start a special container where PID 1 is systemd, and it behaves like a small server.

Unit file: describing a service

The thing systemd manages is called a unit. The most common type is .service. It's a text file with three sections:

# /etc/systemd/system/myapp.service
[Unit]
Description=My application
After=network.target

[Service]
ExecStart=/usr/local/bin/myapp --port 8080
Restart=on-failure
User=myapp

[Install]
WantedBy=multi-user.target

• [Unit]: description and links to other units. After=network.target only sets order: start after the network subsystem. It doesn't promise the network is actually ready, more on that below in the section on dependencies.
• [Service]: how to start it. ExecStart is the command with a full path to the program. Restart=on-failure brings the service back if the process crashes. User sets which user the process runs as. Without this line the service runs as root.
• [Install]: where to hook the service in on enable. WantedBy=multi-user.target means "start on a normal server boot".

You put your own files in /etc/systemd/system/. Files from packages live in /usr/lib/systemd/system/, and a file with the same name in /etc overrides them. Which file systemd loads a service from, and what's in that file right now, is shown by systemctl cat:

systemctl cat myapp

# /etc/systemd/system/myapp.service
[Unit]
Description=My application
...

systemctl: managing services

Every operation goes through systemctl:

systemctl start myapp      # start now
systemctl stop myapp       # stop
systemctl restart myapp    # restart
systemctl status myapp     # state and the last log lines
systemctl enable myapp     # add to autostart
systemctl disable myapp    # remove from autostart

start and enable do different things. start runs the service right now, but it won't come back after a reboot. enable sets autostart, but doesn't start anything right now. Under the hood, enable creates a link in the target directory from WantedBy:

systemctl enable myapp

Created symlink /etc/systemd/system/multi-user.target.wants/myapp.service → /etc/systemd/system/myapp.service.

Usually you need both, and systemctl enable --now myapp does them in one command. Quick checks for scripts: systemctl is-active myapp prints active or inactive/failed, systemctl is-enabled myapp prints enabled or disabled.

daemon-reload

systemd reads unit files into memory and works from that copy. After you edit a file, you need to tell it to reread the configs:

systemctl daemon-reload

Without daemon-reload, restart starts the service from the old version of the file. That's the usual "I fixed it, and nothing changed" complaint. systemctl warns about this, but the line is easy to miss:

Warning: The unit file, source configuration file or drop-ins of myapp.service changed on disk. Run 'systemctl daemon-reload' to reload units.

You can see the same thing in the NeedDaemonReload property: yes means the file on disk is newer than the loaded copy.

systemctl show -p NeedDaemonReload myapp

NeedDaemonReload=yes

systemctl status

systemctl status shows whether the service is alive, where its file was loaded from, and whether autostart is on:

systemctl status myapp

● myapp.service - My application
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; preset: enabled)
     Active: active (running) since Fri 2026-10-09 08:56:37 UTC; 1s ago
   Main PID: 110 (myapp)
      Tasks: 1 (limit: 9281)
     Memory: 8.7M ()
     CGroup: /system.slice/myapp.service
             └─110 /usr/bin/python3 /usr/local/bin/myapp --port 8080

Oct 09 08:56:37 srv systemd[1]: Started myapp.service - My application.

How to read it:

• Loaded: the path to the unit file and the autostart state: enabled or disabled. preset is the distribution's default value; it doesn't affect how the service runs.
• Active: active (running): the service is running. inactive (dead): stopped. failed: crashed; the reason is in the lines below and in the log.
• Main PID: the service's main process. CGroup shows every process in the service; systemd tracks them as a group.
• At the bottom, the last log lines. The full service log is shown by journalctl -u myapp, which we cover in the next lesson.

Here's what a crashed service looks like. ExecStart here points to a file that doesn't exist:

× badpath.service
     Loaded: loaded (/etc/systemd/system/badpath.service; static)
     Active: failed (Result: exit-code) since Fri 2026-10-09 08:56:42 UTC; 986ms ago
    Process: 209 ExecStart=/usr/local/bin/nonexistent (code=exited, status=203/EXEC)

status=203/EXEC means systemd couldn't start the program itself: no file, no execute permission, or an error in the first line of a script. It never got to the app's own code. A normal program exit code looks like status=1/FAILURE.

Restart after a crash

Restart=on-failure brings the service back if the process exited with a nonzero code or was killed by a "dirty" signal like SIGKILL. Easy to test: kill the main process with kill -9, and within a fraction of a second the service has a new Main PID, and the restart counter goes up:

systemctl show -p NRestarts myapp

NRestarts=1

A plain kill without -9 sends SIGTERM, and systemd treats that as a clean shutdown: with on-failure the service stays inactive (dead). Restart=always brings the service back after any exit. systemctl stop never triggers a restart under any setting: stopping it by hand doesn't count as a crash.

Restarts have a safety valve. By default systemd allows 5 starts in 10 seconds. If the service crashes right after starting (say, the port is already taken), the limit runs out, the service stays failed, and the log shows:

myapp.service: Start request repeated too quickly.

Once the cause is fixed, clear the counter and start again:

systemctl reset-failed myapp

systemctl start myapp

Dependencies and order

systemd builds its own graph of what depends on what and in what order to start things. Order and dependency are set by different directives, and that's the main thing to remember.

• After= only sets order: if both units start, this one starts after the one named. It doesn't start the named unit itself.
• Wants= is a soft dependency: when this unit starts, systemd tries to start the named one too, but if that one fails to come up, ours starts anyway.
• Requires= is a hard dependency: if the named unit fails to start (and there's an After= on it), ours doesn't start either. If the named unit is stopped or restarted through systemctl, ours stops or restarts with it.

There's a trap with Requires=. If the dependency crashes on its own (the process exits with an error) and nobody stops it, our service keeps running anyway: Requires= doesn't touch an already running unit when its dependency goes down on its own, rather than through systemctl stop. To make a service stop along with a crashed dependency, you need BindsTo=.

The target multi-user.target is the set of services for a normal server boot. Your service with WantedBy=multi-user.target gets picked up automatically after enable.

There's a separate catch about networking that trips people up a lot. network.target doesn't promise the interface already has an address: it only marks that the network subsystem has started. A service that needs to bind to a specific IP or reach out at startup, with only this dependency, fails with cannot assign requested address. For real connectivity, use a different target:

[Unit]
Wants=network-online.target
After=network-online.target

Wants= is required here: without it, network-online.target might not even make it into the boot plan, and After= would be useless on its own.

Common mistakes

• I fixed the file, and nothing changed. You forgot systemctl daemon-reload. Check the warning in systemctl status output or systemctl show -p NeedDaemonReload.
• The service is gone after a reboot. You ran start without enable. Check systemctl is-enabled.
• enable is done, but the service isn't running. enable without --now starts nothing until the next boot.
• status=203/EXEC. Wrong path in ExecStart, no execute permission, or a broken shebang. Check the path with ls -l.
• The service crashes with Address already in use. An old copy started by hand through nohup is holding the port. Find it with ps aux, stop it, then systemctl reset-failed and start.
• The service runs as root. The [Service] section has no User= line. Create a separate user for each app.
• Requires= didn't stop the service when the dependency crashed. That's by design: that's what BindsTo= is for.

What to remember

systemd is PID 1, and it starts and keeps services running. A service is described by a unit file with [Unit], [Service], and [Install] sections, and your own files live in /etc/systemd/system/.

start runs it now, enable sets autostart, enable --now does both. After editing a unit file, daemon-reload is required.

systemctl status shows the state (active, failed, inactive), the file path, autostart, and the last log lines.

In the next lesson we'll cover where systemd stores service logs and how to read them with journalctl. The "Linux: boot" track covers where systemd sits in the boot chain and what a target is.

What you'll need for the tasks

• Enter the "server" with docker exec -it <name> bash and exit with exit.
• Look at processes: ps aux, stop a process: kill <PID>.
• Write and edit a unit file in /etc/systemd/system/ with the nano editor (save with Ctrl+O and Enter, exit with Ctrl+X).
• The commands systemctl status, start, restart, enable, daemon-reload, reset-failed, systemctl cat, and journalctl -u <service>.
• Check the service from inside: curl -s http://127.0.0.1:<port>/.