systemd as init: targets and boot order

Why this matters

The server rebooted, and your service didn't come up, even though you ran systemctl enable. Or a service only fails at boot, while starting it by hand works on the first try. Both stories come down to how systemd decides what to start and in what order. This lesson covers targets and the dependency graph the boot process follows.

Who is PID 1

By this point the boot chain has already run: firmware -> bootloader -> kernel -> initramfs. The kernel unpacked initramfs into rootfs (ramfs, or tmpfs if the kernel has CONFIG_TMPFS enabled), found the real root, mounted it, and handed control to the first userspace process.

That process is systemd, and it gets PID 1. systemd manages the rest of the daemons. A daemon here means a background service with no terminal: sshd, cron, the network stack. systemd starts them, watches them, and restarts them on failure if the service's settings say to.

ps -p 1 -o pid,comm

    PID COMMAND
      1 systemd

Unit files instead of scripts

systemd's init instructions live in declarative unit files instead of shell scripts. Declarative means the file describes a desired state (what to start, after what, under which conditions), not a top-to-bottom sequence of commands.

One unit describes one entity: a service, a mount point, a socket, a timer or a target. You look at the loaded units like this:

systemctl list-units --type=service
systemctl status ssh

The first command shows active services, the second shows one service's state (on Ubuntu the SSH service is named ssh, on Fedora and RHEL it's sshd). How a unit file itself is structured and what each field means is covered in detail in the linux-admin/systemd-units lesson. Here, two lines from it matter:

• WantedBy=multi-user.target in the [Install] section says which target to attach the service to when you run systemctl enable;
• enable creates a link in the <target>.wants/ directory, for example /etc/systemd/system/multi-user.target.wants/myapp.service.

Targets: system states

A target is a named group of units that systemd brings into an active state. A target is reached once all its dependencies are running. The target itself doesn't do anything; it just groups other units.

Targets you'll run into constantly:

Target	What it means
sysinit.target	early initialization: local filesystems, swap, udevd, tmpfiles
basic.target	sockets, timers, paths: the foundation for services
multi-user.target	server mode, services with no GUI
graphical.target	everything from multi-user plus a graphical login

default.target sets which target the boot starts from. It's a link to one of the targets, and you check it like this:

systemctl get-default

multi-user.target

You can change the default target with systemctl set-default <target>. It rewrites the /etc/systemd/system/default.target link.

Loaded targets:

systemctl list-units --type=target

  UNIT                   LOAD   ACTIVE SUB    DESCRIPTION
  basic.target           loaded active active Basic System
  ...
  multi-user.target      loaded active active Multi-User System
  ...
  sysinit.target         loaded active active System Initialization
  ...

The key consequence for services. At boot, systemd only starts what the default target pulls in, directly or through other targets. If a service is attached to a target the boot doesn't reach, it stays inactive after a reboot, even though systemctl is-enabled will honestly say enabled.

Order comes from the graph, not a queue

graphical.target pulls in multi-user.target, which pulls in basic.target, which pulls in sysinit.target. A graphical session won't come up before the basic initialization does.

systemd has two independent kinds of relations between units:

• requirements (Wants=, Requires=) answer the question "what to start along with me";
• ordering (After=, Before=) answers the question "what to wait for".

A requirement by itself doesn't set an order. If two units only have Wants= or Requires= between them, systemd starts them at the same time. The chain of targets above is ordered because each target has both kinds of relations: for example, multi-user.target has both Requires=basic.target and After=basic.target. You can check this like so:

systemctl show -p Requires -p After multi-user.target

Anything not connected through After=/Before= starts in parallel as soon as its own conditions are met. The order between two independent units isn't defined: one might come up first today, and the other tomorrow.

systemctl list-dependencies graphical.target

graphical.target
● ├─...
● └─multi-user.target
●   ├─...
●   ├─basic.target
●   │ ├─...
●   │ ├─sysinit.target
...

The indentation in the output shows who pulls in whom, not start time. Units at the same level start in parallel if there's no After= between them.

How long the boot took

systemd-analyze

Startup finished in 2.1s (kernel) + 4.8s (userspace) = 6.9s
graphical.target reached after 4.7s in userspace.

This is an example from an ordinary machine. The numbers on yours will differ.

Two more commands show which units held up the boot:

• systemd-analyze blame lists units by start time, the slowest ones on top;
• systemd-analyze critical-chain shows the chain that determined the final time. The number after @ is the moment the unit became active, and after + is how long it took to start.

Targets and runlevels

A target solves the same problem as a runlevel in SysV init: which mode the system runs in. But it's built differently. A runlevel was a number from 0 to 6 with a fixed set of scripts. A target is a node in the dependency graph, there can be as many targets as you like, and you describe your own target with a unit file.

For compatibility, systemd has alias targets: runlevel3.target points to multi-user.target, runlevel5.target to graphical.target.

You can switch a running system to another target with isolate. It starts everything the new target needs and stops everything it doesn't:

systemctl isolate multi-user.target

isolate only works on targets that have AllowIsolate=yes. multi-user.target, graphical.target, and rescue.target have it, basic.target and sysinit.target don't. On a server with an open SSH session, isolate rescue.target will stop sshd too, so only try it in a lab or on a machine with console access.

Common mistakes

• Thinking targets run strictly one after another, like steps, and so does everything inside them. In reality systemd only orders what's connected through After=/Before=, and starts everything else in parallel.
• Writing only Wants= or Requires= and expecting the service to wait for the dependency. Without After= they start at the same time.
• Seeing enabled and deciding a service will definitely come up after a reboot. enabled only means "a link to it exists in some target's .wants/ directory". The service comes up if the boot reaches that target.
• Setting graphical.target as the default on a server "to make sure everything starts". A server with no GUI is fine with multi-user.target, and services need to be attached to a target the boot actually reaches.
• Treating a target and a runlevel as the same thing. A similar idea, but there can be as many targets as you like, and they're connected by a graph, not a number.

What to remember

The first userspace process is systemd with PID 1. Init instructions live in declarative unit files.

A target is a named group of units. multi-user.target has no GUI, graphical.target has one. systemctl get-default shows the default target.

Boot follows the graph. Requirements (Wants=, Requires=) say what to start, ordering (After=, Before=) says what to wait for. Anything not ordered starts in parallel.

The next lesson covers udev: how the system discovers devices and populates /dev.

What you'll need for the tasks

• Log into the "server" with docker exec -it <name> bash and exit with exit. Reboot it from outside: docker restart <name>.
• systemctl get-default, systemctl set-default, systemctl list-dependencies <target>.
• systemctl status <service>, systemctl cat <service>, systemctl is-enabled <service>, systemctl show -p <Property> <unit>.
• systemctl enable, systemctl disable, systemctl daemon-reload after editing a unit file.
• The current boot's journal for a service: journalctl -b -u <service>.
• Edit files in /etc/systemd/system/ with the nano editor (save with Ctrl+O and Enter, exit with Ctrl+X).