udev: how the system finds devices

Why this matters

You plug in a USB adapter, and the program can't open the port: Permission denied. Or after a reboot the second disk becomes /dev/sda, and the server fails to mount the partition from /etc/fstab. Both stories are about the same link in the boot chain: how devices show up in /dev, who gives them permissions, and where stable names come from. That's udev's job.

Where we are in the boot chain

By this point the system has gone through firmware -> bootloader -> kernel -> initramfs. On UEFI there may be no separate bootloader at all: the firmware launches a kernel with an EFI stub directly. Then systemd worked through its targets by dependency (graphical -> multi-user -> basic -> sysinit), and the system came up.

What's left is the hardware. Disks, network cards, USB sticks, sound chips: programs talk to them through device nodes in the /dev directory. A node looks like a file, but reads and writes to it go to a driver.

Who populates /dev

The kernel creates the nodes themselves. The /dev directory is mounted as devtmpfs: an in-memory filesystem where the kernel keeps a node for every registered device. The name and permissions on such a node are the defaults; the kernel knows nothing about your groups and users.

mount | grep ' /dev '

devtmpfs on /dev type devtmpfs (rw,nosuid,relatime,size=...)

udev runs on top of devtmpfs. udev is a userspace daemon (a background process) that receives device events from the kernel, sets permissions and ownership on nodes, adds human-readable symlinks, and renames network interfaces. In systemd it lives as the systemd-udevd service.

systemctl status systemd-udevd

● systemd-udevd.service - Rule-based Manager for Device Events and Files
     Loaded: loaded (/usr/lib/systemd/system/systemd-udevd.service; static)
     Active: active (running) since Sat 09:00:04 UTC; 2h ago

Look at the Active line: this is a regular process with a PID, not code inside the kernel. The kernel only reports events; udev makes the decisions in user space.

How it works

When the kernel discovers a device (a flash drive gets plugged in, a network card comes up), it sends a uevent. This is a message that a device has appeared, disappeared, or changed. udev listens to this stream of events.

On receiving a uevent, udev runs the device through the udev rules: text files that describe what to do with the device. Rules set permissions, owner, group, extra symlinks, and variables for other programs.

You can watch the stream of events with udevadm monitor. Here's roughly what you see when a disk gets connected:

udevadm monitor --udev

monitor will print the received events for:
UDEV - the event which udev sends out after rule processing

UDEV  [1204.512301] add      /devices/.../block/sdb (block)
UDEV  [1204.534118] add      /devices/.../block/sdb/sdb1 (block)

Each line: a timestamp, an action (add), the device path in /sys, and the subsystem in parentheses. Exit the monitor with Ctrl+C.

udev rules

System rules live in /usr/lib/udev/rules.d/ (/lib/udev/rules.d/ on older systems); your own go into /etc/udev/rules.d/. udev reads only files with the .rules extension; a file named 70-my.rule or 70-my.rules.bak is silently skipped.

udev sorts all files from all directories together by name and reads them in order. That's why the number at the start of the name matters: 80-lab.rules from /etc is read after 60-block.rules from /usr/lib and before 90-iocost.rules. If /etc/udev/rules.d/ has a file with the same name as a system one, it replaces the system file entirely.

Each rule line is a set of keys separated by commas. Some keys are conditions, some are actions:

SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", MODE="0660", GROUP="dialout", SYMLINK+="ftdi0"

A line from /etc/udev/rules.d/70-usb-serial.rules for an FTDI USB adapter. Keys with == (or !=) compare: the device's subsystem is tty, and the parent USB device has this vendor and model. Keys with = and += assign: mode 0660, group dialout, one more symlink /dev/ftdi0 added to any that already exist. The line fires only if every condition matches.

ATTRS{...} looks for an attribute not just on the device itself but on every parent up the /sys tree. The tty node has no idVendor attribute; it belongs to the USB device this port is part of. ATTR{...} without the S looks only at the device itself.

A mixed-up operator breaks the whole line. udev won't accept KERNEL="sdb" (an assignment instead of a comparison); it logs an error and skips the entire line. You can check a file before applying it with udevadm verify (available in systemd 254 and newer):

udevadm verify /etc/udev/rules.d/70-usb-serial.rules

1 udev rules files have been checked.
  Success: 1
  Fail:    0

After editing rules, reload them and replay the events for devices that are already connected:

udevadm control --reload

udevadm trigger

Without trigger, the new rules only apply to devices that appear later.

Rules have a separate job too: network interface names. The name eth0 depends on the order in which the kernel found the cards. udev's built-in net_id module computes a name from the card's position on the bus, hence enp3s0 (Ethernet, PCI bus 3, slot 0). The 80-net-setup-link.rules rule applies it.

Inspecting a device

To write a condition, you need to know what udev sees on a device. udevadm info shows this:

udevadm info /dev/sdb1

P: /devices/pci0000:00/.../block/sdb/sdb1
M: sdb1
U: block
N: sdb1
S: disk/by-uuid/e5f6a7b8-1234-5678-9abc-def012345678
E: DEVNAME=/dev/sdb1
E: SUBSYSTEM=block
E: ID_FS_TYPE=ext4
E: ID_FS_UUID=e5f6a7b8-1234-5678-9abc-def012345678

P: is the device path in /sys, N: is the node name in /dev, U: is the subsystem (checked by SUBSYSTEM==). S: lines are symlinks udev created. E: lines hold variables that rules can check with conditions like ENV{ID_FS_TYPE}=="ext4".

udevadm info -a prints attributes for ATTR and ATTRS across the whole chain of parents. It prints them already in rule format, so you can copy conditions straight from it:

udevadm info -a /dev/sdb1

  looking at device '/devices/pci0000:00/.../block/sdb/sdb1':
    KERNEL=="sdb1"
    SUBSYSTEM=="block"
    ATTR{size}=="2097152"
...
  looking at parent device '/devices/pci0000:00/.../block/sdb':
    KERNELS=="sdb"
    SUBSYSTEMS=="block"
...

udevadm test lets you check a rule without a real event. It runs the rules for a device and prints which line in which file did what, and what the resulting variables are:

udevadm test --action=add /sys/class/block/sdb1

...
sdb1: /etc/udev/rules.d/80-backup.rules:1 Added SYMLINK 'backup-disk'
...
DEVNAME=/dev/sdb1
SUBSYSTEM=block
DEVLINKS=/dev/backup-disk /dev/disk/by-uuid/e5f6a7b8-...

The command is meant for debugging and does not run the programs from RUN keys. It does apply permissions and symlinks for real, though, so run it as root and only on your own machine.

by-uuid and by-id symlinks

A disk's name (/dev/sda, /dev/sdb) depends on discovery order and can change between reboots. That's why udev creates stable symlinks through the 60-persistent-storage.rules rules: by the filesystem's UUID, by model and serial number, by position on the bus.

ls -l /dev/disk/by-uuid/

lrwxrwxrwx 1 root root 10 Oct 12 09:00 a1b2c3d4-... -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 12 09:00 e5f6a7b8-... -> ../../sdb1

Next to it you'll find /dev/disk/by-id/ (model and serial number) and /dev/disk/by-path/ (controller port). /etc/fstab uses UUID=...: it doesn't depend on which disk was probed in what order.

Common mistakes

• Thinking files in /dev live on disk. /dev is devtmpfs in memory; the kernel repopulates it on every boot. A node you created by hand with mknod or changed with chmod disappears after a reboot or when the device reconnects. Set permanent permissions and names with a rule.
• Thinking udev is part of the kernel. The kernel creates the node and sends the uevent, while a separate systemd-udevd process sets permissions, groups, and symlinks.
• Naming a file something other than .rules. udev simply doesn't read it and says nothing.
• Writing = in a condition. The line with the error gets dropped entirely, and udevadm verify will show it.
• Writing a condition by guesswork. Pull values for SUBSYSTEM, KERNEL, ATTR from udevadm info, not from someone else's rule on a forum: your device's subsystem might be named differently.
• Fixing a rule and expecting results right away. Without udevadm control --reload and udevadm trigger, already-connected devices won't notice anything.

What you'll need for the tasks

• udevadm info <node>: the U: line (subsystem) and N: line (node name), and udevadm info -a for keys in rule format.
• Rule line syntax: conditions with ==, actions with = and +=, the keys SUBSYSTEM, KERNEL, MODE, GROUP, SYMLINK.
• Where your own rules live and what extension the file needs.
• udevadm verify <file> to check syntax, and udevadm test --action=add <path in /sys> to run the rules. In the test output, look for lines with your file's name and the DEVLINKS= line.
• In the labs, udev runs inside a container without real hardware and without a running systemd-udevd, so udevadm control --reload and udevadm trigger don't work there. We check the result of rules with udevadm test.