Process limits: ulimit, limits.conf, and systemd

Why limits exist at all

One misbehaving process shouldn't be able to take down the whole server: eat all the memory, open files without end, spawn endless processes. That's why the kernel keeps resource limits on every process. It's a safety fuse.

Under load, these same limits sometimes get in the way. A service legitimately needs a lot of open connections, and the default limit is too small. Then you raise the limit on purpose, in the right place. Which place that is depends on how the process was started, and that's where most people trip up.

ulimit: look and change

The built-in bash command ulimit shows the current limits of the shell and everything started from it. The limit on open files (descriptors) is checked with the -n flag:

ulimit -n

1024

ulimit -a shows all the limits at once:

ulimit -a

...
max locked memory           (kbytes, -l) 64
open files                          (-n) 1024
...
max user processes                  (-u) unlimited
...

-n comes up more often than the rest in practice: it's the maximum number of open descriptors per process. A loaded web server or database can easily run through 1024, and that's when you get Too many open files from the previous lesson. Every connection, file, and pipe takes up one descriptor.

Soft and hard

Every limit has two boundaries:

• soft — the value that actually applies. This is what the process hits. A process can raise it itself, but not above hard.
• hard — the ceiling soft can't go above.

ulimit -Sn

1024

ulimit -Hn

524288

An ordinary user can raise their soft limit up to hard, but not above the ceiling the admin set:

ulimit -Sn 4096

Hard has a catch. Any process can lower it, with no special permission, and there's no going back: in that process and its children, the new ceiling is final. Only root, or more precisely a process with the CAP_SYS_RESOURCE capability, can raise hard back up.

ulimit -Hn 4096
ulimit -Hn 65535

bash: ulimit: open files: cannot modify limit: Operation not permitted

The first command went through silently, the second didn't.

Now the trap. ulimit -n 65535 without -S or -H changes both boundaries at once: soft and hard both become 65535. If hard was 524288, you just lowered it irreversibly for this shell. That's why in scripts you write ulimit -Sn 65535 explicitly.

ulimit -n 2048
ulimit -Sn
ulimit -Hn

2048
2048

One more mistake in the same family: soft can't be set above hard, bash will answer cannot modify limit: Invalid argument.

The limit lives in the process

Limits belong to the process, not to the user or the system. A child gets a copy of the parent's limits the moment it starts. Two things follow from this:

• ulimit in your terminal only affects that shell and whatever you launch from it. A service that's already running won't notice.
• The change lasts until the end of the session. Close the terminal and it's gone.

To find out the actual limit of a running process, look at /proc/<PID>/limits. It's the only honest answer; everything else is just configuration:

grep 'open files' /proc/812/limits

Max open files            1024                 524288               files

Columns left to right: soft, hard, units. prlimit from util-linux gives the same answer, and it can also change the limit of someone else's running process without a restart (root):

prlimit --pid 812 --nofile

RESOURCE DESCRIPTION              SOFT   HARD UNITS
NOFILE   max number of open files 1024 524288 files

prlimit --pid 812 --nofile=4096:8192

This is an emergency measure. After a restart, the process gets its limits from its configuration again.

limits.conf: login limits

Persistent limits for people are set in /etc/security/limits.conf or in a separate file under /etc/security/limits.d/ (the filename must end in .conf). A line has four fields: who, which boundary, which resource, the value.

# /etc/security/limits.conf
www-data  soft  nofile  65535
www-data  hard  nofile  65535
@backend  -     nofile  20000

• First field: the username, @name for a group, * for everyone.
• Second: soft, hard, or - (both boundaries at once).
• Third: the resource. nofile is open files, nproc is the number of processes. An unknown resource name silently cancels the line.
• Fourth: the value.

This file is read by the PAM module pam_limits, and only at login: login, SSH, su, runuser, and on Ubuntu also cron. Two important things follow. Only a new session gets the new limits: a terminal that's already open keeps the old ones. Check it like this:

su - www-data -s /bin/bash -c 'ulimit -Sn; ulimit -Hn'

65535
65535

And the big one: processes started without logging in through PAM never see this file at all. That includes systemd services, containers, and docker exec.

systemd: service limits

A systemd service never "logs in", so limits.conf doesn't apply to it. The limit is set in the unit file, in the [Service] section:

[Service]
LimitNOFILE=65535

A single number sets soft and hard to the same value. A colon lets you set them separately: LimitNOFILE=4096:65535. If the unit has nothing set, systemd's default applies (DefaultLimitNOFILE=). On systemd 255 on Ubuntu 24.04 that's soft 1024 and hard 524288.

systemctl show tells you what systemd actually applies to the service:

systemctl show -p LimitNOFILE -p LimitNOFILESoft myapp

LimitNOFILE=65535
LimitNOFILESoft=65535

Here LimitNOFILE is hard, LimitNOFILESoft is soft. After editing you need systemctl daemon-reload and systemctl restart myapp: the limit is handed to the process at startup, a running process doesn't re-read it.

A unit isn't built from a single file. There can be a myapp.service.d/ directory next to it with additions (a drop-in), for example override.conf, which systemctl edit myapp creates. Settings from a drop-in are read after the main file and override it. If the main unit has LimitNOFILE=65535 and the drop-in has LimitNOFILE=1024, the service gets 1024. systemctl cat myapp shows the whole unit together with its drop-ins, and systemctl show shows the final value.

And in a container

A container has its own limits, set by Docker at startup with the --ulimit flag:

docker run --rm --ulimit nofile=4096:8192 ubuntu:24.04 bash -c 'ulimit -Sn; ulimit -Hn'

4096
8192

Root inside a container doesn't have CAP_SYS_RESOURCE by default, so it can't raise hard above what Docker gave it. You raise it with docker run --ulimit or in compose (ulimits:).

Common mistakes

• Edited limits.conf for a service. A systemd-managed service doesn't read it. Checking with su - user -c 'ulimit -n' will show the new value and mislead you: su goes through PAM, the service doesn't. Look at /proc/<PID>/limits of the process itself.
• Added LimitNOFILE= and didn't restart. Without daemon-reload systemd doesn't know about the edit, without restart the process keeps living with the old limit.
• Missed a drop-in. You edit the main unit, and a file in *.service.d/ overrides the value. systemctl cat shows all the files.
• ulimit -n instead of ulimit -Sn in a startup script. This also lowers hard, and this process can never raise it again.
• A typo in limits.conf. nofiles instead of nofile, or devs instead of @devs: the line doesn't work, and you won't see an error on screen.
• Checked the wrong process. ulimit -n in your own terminal talks about your shell, not about the service.

What you'll need for the tasks

• ulimit -Sn, ulimit -Hn, and how they differ;
• /proc/<PID>/limits for a running process, the service's PID from systemctl show -p MainPID --value <service>;
• the format of /etc/security/limits.conf and limits.d/: @group, soft/hard, nofile;
• su - <user> -c 'ulimit -Sn; ulimit -Hn' to check a new session;
• LimitNOFILE=, drop-ins in *.service.d/, systemctl cat, systemctl show, daemon-reload, restart;
• for the mini-project, everything from the module: systemctl and journalctl (lessons 1-2), cron (lesson 4), EnvironmentFile (lesson 5), df/du and deleted open files (lessons 6 and 9), jq (lesson 7).

What to remember

Resource limits keep one process from taking down the server. The one that matters most in practice is nofile (open files). The limit is double: soft applies, hard caps soft. Anyone can lower hard, and it's irreversible; only root with CAP_SYS_RESOURCE can raise it.

The limit lives in the process and is inherited by its children. For people, it's set in limits.conf (applies at login), for systemd services through LimitNOFILE= in the unit, for containers through --ulimit. Always check /proc/<PID>/limits of the process that's actually failing.

This is the last lesson of the linux-admin module. In the mini-project you'll hand over a server after a move and use everything at once: services, the journal, environment variables, disk, cron, and limits.