Check: backups

RAID covers exactly one scenario: a disk failure. It duplicates logical errors across every disk in the array, because to RAID this is just an ordinary write. Parity doesn't hold a previous version of the data, it holds a sum of the current blocks, which is used to rebuild a dead disk. More in the lesson RAID: what it actually gives you.

Data is lost from 03:00 to 14:00, that's 11 hours of work. A once-a-day schedule guarantees an RPO no better than 24 hours, but in any specific incident the loss equals the time elapsed since the last copy. And that's assuming the last copy is actually working. If the nightly backup silently broke, you need to count from the one before it, and that's exactly what the mini-project below is about.

The extra "one" means an immutable or offline copy. The "zero" means no errors during verification: a copy only counts as a copy once it has passed a successful check. The scheme is covered in the lesson The 3-2-1 rule.

Because of this, restoring from incrementals needs the whole chain in order, while a differential only needs two archives: the full one and the latest differential. One broken link in an incremental chain cuts off every day after it.

A second snapshot writes only the new blocks to the repository. At the same time, each snapshot stays self-contained: you can restore any of them, there's no chain to assemble. In the lesson restic in practice, the second backup processed the same 190 KiB and added about 2 KiB.

Check it yourself. The command backs up a megabyte of random data, appends a small file, and backs up again:

docker run --rm --entrypoint sh restic/restic:0.19.1 -c 'export RESTIC_PASSWORD=lab-only-password RESTIC_REPOSITORY=/repo; restic init -q; mkdir /data; head -c 1048576 /dev/urandom > /data/photos.bin; restic backup -q --host demo /data; echo "one more line" >> /data/notes.txt; restic backup --host demo /data | grep -E "^(Files|Added|processed)"'

Files:           1 new,     0 changed,     1 unmodified
Added to the repository: 1.116 KiB (800 B stored)
processed 2 files, 1.000 MiB in 0:00

A megabyte was processed, but only about a kilobyte was added: the new file and metadata.

Space for changes is allocated when the snapshot is created (lvcreate -s -L), but it fills up gradually: old versions of blocks that change on the source volume get copied there. Running out of space makes a classic snapshot invalid. A snapshot lives on the same disks as the volume, so it doesn't protect against their failure. By itself it also doesn't give you application-level consistency: a database sees such a snapshot as the state after a sudden power loss.

This is PITR, point-in-time recovery. You restore the base backup, then replay the WAL logs on top of it up to the moment you need. The restore parameters (restore_command, recovery_target_time) are set in postgresql.conf, and a recovery.signal file is created in the data directory. A logical dump freezes a single moment, you can't replay logs on top of it.

By default restic groups snapshots by host and paths (--group-by host,paths), and the retention rule works inside each group. Snapshots from different hosts form different groups. This behavior is changed with the --group-by flag, and cleaning up only your own snapshots is set with --host.

Check it yourself on four snapshots from two hosts. --dry-run only shows what would be removed:

docker run --rm --entrypoint sh restic/restic:0.19.1 -c 'export RESTIC_PASSWORD=lab-only-password RESTIC_REPOSITORY=/repo; restic init -q; mkdir /data; echo a > /data/f; for h in web-01 web-01 web-02 web-02; do restic backup -q --host $h /data; done; restic forget --keep-last 1 --dry-run | grep -E "^(keep|remove) "; echo ---; restic forget --keep-last 1 --group-by "" --dry-run | grep -E "^(keep|remove) "'

keep 1 snapshots:
remove 1 snapshots:
keep 1 snapshots:
remove 1 snapshots:
---
keep 1 snapshots:
remove 3 snapshots:

With the default grouping, --keep-last 1 kept one snapshot per host. With --group-by "" the whole repository became one group, and only one snapshot remained in total.

Replication repeats every change, including mistakes: a DELETE without a WHERE reaches the replica within milliseconds. A middle ground is a delayed replica with the recovery_min_apply_delay parameter: it applies changes with a delay, giving you time to pull data off it. Covered in the lesson What a backup is not.