Service logs: journald and journalctl

A service crashed, systemctl status showed the last few lines, and they don't say why. You need the service's full log next. On a systemd host it lives in the journald journal, and you need to read it fast: filtered by service, by time, and by severity.

Where service logs go

In the past, each service wrote to its own file in /var/log/, and you had to remember a dozen paths. On a systemd host, one component collects every service's logs: journald (the systemd-journald service). It takes in each unit's stdout and stderr, messages sent over the syslog socket (for example, from the logger command), and systemd's own messages. For every entry it adds metadata: which unit, which process, when, and at what priority. All of it goes into one binary journal.

You can't read this journal with cat or grep; there's a dedicated command, journalctl, for that. In exchange, it can filter by unit, time, and priority, which plain grep over flat files can't do painlessly.

The root user reads the whole journal. A regular user only sees their own entries in it; to see everything, they're added to the systemd-journal group (the whole journal is also readable by the adm and wheel groups). That's why the commands below run through sudo on a server. A user without access to the system journal gets:

No journal files were opened due to insufficient permissions.

journalctl: basic reading

journalctl                 # the whole journal, oldest to newest
journalctl -e              # jump straight to the end (the freshest entries)
journalctl -n 50           # the last 50 lines
journalctl -r              # reverse order, newest first

In a terminal, the output opens in the less pager: scroll with the arrow keys and space, search with /, quit with q. If the output goes to a pipe or a file, there's no pager. You can turn it off explicitly with --no-pager. In a terminal, lines are colored by priority: err and above are red, warning is yellow.

Most of the time you don't want the whole log, just one service's.

Filtering by unit

journalctl -u myapp              # only logs from unit myapp
journalctl -u myapp -u nginx     # several units at once

-u (unit) is the main filter you'll use day to day. It's the same unit that systemctl manages, from the previous lesson. The output looks like this:

journalctl -u myapp -n 4

Oct 09 09:44:28 srv systemd[1]: Started myapp.service.
Oct 09 09:44:29 srv demo[89]: listening on :8080
Oct 09 09:44:29 srv demo[89]: ERROR payment failed: gateway timeout
Oct 09 09:44:29 srv demo[89]: payment failed: gateway timeout (prefixed)

Each line has a date, the hostname, the process name with a PID in brackets, and the message itself. Notice that the program's name (demo) is in the middle, not the unit's name. Here, the unit myapp.service runs the file /usr/local/bin/demo. The line systemd[1]: Started ... also made it into the output: -u shows systemd's own messages about this unit too.

If the unit name has a typo, journalctl doesn't complain; it just prints -- No entries -- and exits successfully. For empty output, first check with systemctl status <name> whether the unit even exists.

You can watch the log in real time, similar to tail -f:

journalctl -u myapp -f

-f (follow) keeps the output open and appends new lines as they appear. This is how you watch a service during a deploy or while debugging. Exit with Ctrl+C.

Two more flags you'll often use alongside -u:

• -g <pattern> (grep) keeps only entries whose message matches a regular expression: journalctl -u myapp -g timeout.
• -o changes the output format. -o short-iso prints the date in ISO format (2026-10-09T09:44:29+00:00), -o cat prints only the message text.

Filtering by time

journalctl -u myapp --since "2026-06-18 09:00"
journalctl -u myapp --since "1 hour ago"
journalctl --since today
journalctl --since "10 min ago" --until "5 min ago"

--since and --until take both absolute and relative time (yesterday, 2 hours ago). This cuts out noise right away: you look at the window around an incident, not the whole journal. If there's nothing in the window, you get -- No entries --.

A separate filter, -b, shows entries from a single system boot: -b for the current one, -b -1 for the previous one. journalctl --list-boots prints the list of boots. This only works if the journal is stored on disk (more on that below).

Filtering by priority

Every entry has a priority, like in syslog. The -p flag shows entries at that level and above:

journalctl -u myapp -p err     # only errors and worse
journalctl -p warning          # warning and up, across all units

Levels from quietest to loudest: debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), emerg (0). You can use a number instead of a name: -p 3 is the same as -p err. When debugging an outage, people usually start with -p err to see the errors right away.

There's a trap here. Everything a service writes to stdout and stderr is recorded by journald at info priority by default. The word ERROR in the text of a line doesn't affect its priority. In the example above, the line ERROR payment failed doesn't show up under -p err:

journalctl -u myapp -p err

Oct 09 09:44:29 srv demo[89]: payment failed: gateway timeout (prefixed)

Only the line the program prefixed with <3> made it through. journald reads such a prefix (<0> through <7>) as the entry's priority and strips it from the text. Applications that know about journald do this. You can also set a priority by hand with logger:

logger -p user.err -t billing "card declined"

You can see a specific entry's priority in the verbose format, journalctl -u myapp -o verbose, in the PRIORITY= field. Bottom line: -p works reliably for services that set their own priority. For the rest, search by text with -g.

Where the journal is stored

journald keeps the journal in one of two places:

• /run/log/journal/ lives in RAM and disappears on reboot;
• /var/log/journal/ lives on disk and survives a reboot.

Where it writes is set by the Storage= parameter in /etc/systemd/journald.conf. The default is auto: if the directory /var/log/journal/ exists, the journal is written to disk; if not, only to memory. persistent writes to disk and creates the directory itself, volatile writes only to memory.

It's more convenient to put settings in a separate file under /etc/systemd/journald.conf.d/*.conf instead of journald.conf itself. Such files are read in alphabetical order, and if a setting repeats, the one read last wins. What's actually in effect right now, across all the files in the right order, is shown by:

systemd-analyze cat-config systemd/journald.conf

After changing journald's settings, restart it:

systemctl restart systemd-journald

If the journal was living in memory before this, it won't move to disk right away: a separate service does that on boot. Without a reboot, ask journald to move the entries to disk yourself:

journalctl --flush

Where journald writes and what its limit is, it reports in its own log:

journalctl -u systemd-journald -o cat

Journal started
Runtime Journal (/run/log/journal/851b2d0a7df34d3ea2386f903a586c42) is 8.0M, max 392.5M, 384.5M free.
...
System Journal (/var/log/journal/851b2d0a7df34d3ea2386f903a586c42) is 11.6M, max 50.0M, 38.3M free.

Runtime Journal is the journal in memory, System Journal is the journal on disk. The long string in the path is the server's machine-id.

Journal size

The journal doesn't grow forever. By default journald uses no more than 10% of the filesystem (and no more than 4 GB), evicting old entries. The limit is set by the SystemMaxUse= parameter, for example SystemMaxUse=500M.

To see how much space the journal takes up, and to clean it up by hand:

journalctl --disk-usage            # how much space is used
journalctl --vacuum-time=7d        # delete entries older than 7 days
journalctl --vacuum-size=500M      # keep no more than 500 MB

--vacuum-* only deletes archived journal files, it doesn't touch the current open file. So on a fresh system, vacuum often reports freed 0B. To clean up the current entries too, close the current file first with journalctl --rotate, then run vacuum. A one-off cleanup doesn't replace the limit: without SystemMaxUse= the journal will grow again.

Common mistakes

• -p err is empty even though there's an ERROR in the log. The application writes to stdout with no priority, so everything is logged as info. Search with -g ERROR instead.
• -- No entries -- on journalctl -u. A typo in the unit name, or an empty --since window. Check the name with systemctl status.
• journalctl -p err without -u shows other services' errors. On a server something is always complaining. Narrow it down to your unit first.
• systemctl status doesn't show why it crashed. It only prints a handful of recent lines, often just systemd's own messages that the process exited. Open the full log with journalctl -u.
• No logs after a reboot. The journal lives in /run: either the directory /var/log/journal/ doesn't exist, or Storage=volatile is set somewhere.
• No journal files were opened due to insufficient permissions. You're not root and not in the systemd-journal group. Run it with sudo.

What to remember

journald collects the stdout and stderr of every unit into one journal with metadata, and you read it with journalctl.

The main filters: -u by unit, --since/--until by time, -p by priority, -g by text, -f to follow in real time. You combine them. -p only sees the priority the application itself set.

Storage= decides where the journal is stored: in memory it disappears on reboot, on disk (/var/log/journal/) it stays. SystemMaxUse= caps its size.

Logs come up again in the observability track: log levels, structured logs, and shipping to centralized systems.

What you'll need for the tasks

This lesson's labs run in the same systemd container as the previous lesson: PID 1 there is systemd, and you enter it as root.

• Enter the "server" with docker exec -it <name> bash, exit with exit.
• journalctl -u <unit> with the flags -p, -n, -f, -g, --since.
• systemctl status, systemctl restart, systemd-analyze cat-config.
• File permissions: ls -l, chmod, chown/chgrp from the permissions lesson.
• Edit files with the nano editor (save with Ctrl+O and Enter, exit with Ctrl+X).