Full, incremental, differential backups, and how long to keep them

How restore chains work, how deduplication differs from an incremental, and how the grandfather-father-son retention scheme works.

Three ways to avoid copying everything every time

A full copy of a terabyte database every night means a terabyte of traffic and a terabyte of storage every day. So you don't copy everything.

A full backup contains all the data. It restores from a single archive, and takes the most space.

An incremental backup contains the changes since the last backup of any type. It's the lightest, but restoring it needs the full backup plus the whole chain of incrementals in order.

A differential backup contains the changes since the last full backup. It grows day by day, but restoring it needs only two archives: the full backup and the latest differential.

The cost of the choice shows up at restore time

An incremental scheme: full on Sunday, incrementals on the other days. To restore Saturday you need Sunday's full backup and six incrementals in a row. A broken Tuesday breaks everything after it: Wednesday, Thursday, and beyond can no longer be assembled, leaving only Sunday and Monday.

A differential scheme: full on Sunday, each day a diff from Sunday. To restore you need two archives, and a broken Tuesday only costs you Tuesday.

Incrementals save space and backup time, differentials win on restore time and reliability. The choice depends on what matters more to you, and on your bandwidth.

Which gives a practical rule: an archive sitting on disk isn't a backup yet. A backup is an archive that reads to the end. Check that in advance, not on the day of the incident.

Deduplication: a different approach

Tools like restic, borg, and kopia skip the chain and instead cut data into blocks, storing each unique block once. Classic enterprise backup systems (Veeam, for instance) also deduplicate, but they still keep the full-plus-incrementals chain, so everything said above about a broken link still applies to them.

From the outside, every snapshot in the repository is self-contained: you restore any one of them directly, with no chain to assemble. Internally, identical blocks get reused, so the hundredth snapshot takes roughly as much space as what changed since the ninety-ninth.

This breaks the usual arithmetic. With deduplication, thirty daily snapshots often cost less than one full backup plus incrementals, because repeated data isn't duplicated at all.

The flip side: the repository becomes a linked structure. Damage to one block affects every snapshot that uses it. That's why these tools have a separate integrity-check command: restic check, borg check.

Retention: how long to keep backups

You can't keep everything forever, and you can't delete right away either. The classic scheme is called grandfather-father-son (GFS) and lays copies out in rings. The numbers below are common practice, not a rule, and you tune them for the specific system:

• daily backups kept for 7 days;
• weekly backups kept for 4-5 weeks;
• monthly backups kept for 12 months;
• sometimes yearly backups kept for a few years on top.

This pyramid gives frequent restore points for recent events and sparse ones for old data. You want to roll back yesterday's mistake precisely, and a year-old contract just needs to be findable.

Separately, think about regulatory requirements. In some fields the retention period for certain data is set by law, and it overrides your own storage math.

Corruption you notice a month later

The main argument for a long retention window is silent errors, not outages. A bug in the application corrupted a column for a month and nobody noticed. If you only have a week of copies, all of them already contain the corrupted data.

So your retention depth needs to cover how long it actually takes to detect problems. A week protects you from an outage, months protect you from your own code.

Common mistakes

• Restoring from the full backup and the latest incremental only. The latest incremental holds only one night's changes. Without the in-between links, you get a mix of old data and a single day.
• Applying archives out of order. ls and the *.tar.gz pattern sort alphabetically: inc-fri comes before inc-mon. Set the order explicitly.
• Forgetting about deletions. If the tool can't apply deletions from the chain, files deleted mid-week come back on restore.
• Assuming an archive is fine because it exists. File size and date say nothing about whether it reads to the end. Run integrity checks on a schedule.
• Retention shorter than detection time. A week of copies won't save you from a bug that was noticed a month later.

What you'll need for the tasks

In the labs, GNU tar in --listed-incremental mode (short form -g) builds the chain. That's enough to get hands-on with a chain without a dedicated backup tool.

While creating an archive, tar keeps a snapshot file (usually .snar) tracking what's already been saved and when. The first run with a new snapshot file produces a full archive, later runs with the same file produce incrementals:

tar -czf full-sun.tar.gz -g backup.snar -C /data .
tar -czf inc-mon.tar.gz -g backup.snar -C /data .

A differential archive comes from keeping a copy of the snapshot file saved right after the full backup, and reusing that copy before each run. Then tar always compares the data against the full backup, not against yesterday's.

See what's inside an archive: tar -tvzf <archive>. If the archive is corrupted, the command prints an error like gzip: stdin: unexpected end of file and exits with a nonzero code. To check a compressed file without unpacking it, use gzip -t <archive>.

When unpacking an incremental, tar deletes files that didn't exist at the time the archive was created only if you pass --listed-incremental=/dev/null (or -G). Without this option, the incremental unpacks like an ordinary archive, and deleted files stay put. Details in the GNU tar manual, section Using tar to Perform Incremental Dumps.

What to remember

An incremental holds the difference from the last backup, a differential holds the difference from the last full backup. The first is cheaper, the second is more reliable to restore.

A chain of incrementals restores as a whole: one broken link breaks everything after it.

Deduplication makes every snapshot self-contained on the outside and shared on the inside, so a long history costs little.

The grandfather-father-son scheme gives frequent points for recent data and sparse ones for old data.

You choose retention depth by how long silent corruption takes to detect, not only by outages.

Next, on snapshots: how they differ from a copy, and why consistency matters more than speed.