Images and layers

In the last lesson you ran containers from ready-made images and didn't think about what was inside them. Here we'll look at what an image is made of, why ten containers from one image barely take up extra space, how a tag differs from a digest, and why a tag without an exact version bites you at the worst possible moment. This matters when an image suddenly weighs three times more than expected, or when "the same" version behaves differently on two servers.

What an image is made of

From the outside, nginx:1.27.3-alpine looks like one file. It's actually a manifest plus a list of layers. The manifest is a small JSON describing what's inside. The layers are archives of filesystem changes.

When you run docker pull nginx:1.27.3-alpine, Docker:

1. Fetches the manifest.
2. Downloads, in parallel, only the layers you don't already have.
3. Stores them locally, indexed by hash.

If two images share a base, its layers sit on disk only once. That's why the first download is slow, while the next image on the same base arrives faster: in the docker pull output, such layers say Already exists.

Image layers

The RUN, COPY and ADD instructions each create a new layer, an archive of whatever the instruction added or changed. FROM doesn't add its own layer, it sets the starting stack of layers from the base image. Layers are read-only, identified by a hash of their content, and shared across images.

When you run a container, a single writable layer sits on top of the read-only stack. Every change inside the container is written there. The container's filesystem is assembled from the layers bottom to top, usually through OverlayFS (OverlayFS storage driver).

This is copy-on-write. As long as nobody touched a file, it's one copy shared by every container from this image. The moment someone changes it, a copy moves up into the writable layer of only that container. Neighboring containers still see the original.

What follows from this:

• Running 10 containers from one image is cheap on disk: the base layers aren't duplicated.
• You can change files inside a container, but it's local to that container. Edits survive docker stop and docker start (and docker restart), because the writable layer lives alongside the container. After docker rm and a fresh docker run, they're gone: the new container gets an empty writable layer. We'll cover storing data separately from the container in the lesson about volumes.
• Deleting a heavy file in the next instruction doesn't work. If a RUN downloaded an archive, and the next RUN deleted it, the archive stays in the first instruction's layer, and the image weighs the same. Deleting only hides the file in the top layer.

What changed in the container: docker diff

You can see what's in the writable layer with docker diff. It compares the container's filesystem with the image and marks each line with a letter: A (file added), C (changed), D (deleted) (docker container diff).

docker run -d --name cow-demo --memory 32m --cpus 0.25 alpine:3.20.3 sleep 300
docker exec cow-demo sh -c 'echo changed > /etc/motd; touch /root/new.txt; rm /etc/issue'
docker diff cow-demo

C /root
A /root/new.txt
C /etc
C /etc/motd
D /etc/issue

The /root and /etc directories are marked C because something changed inside them. A fresh container from the same image shows an empty docker diff. Keep in mind that many programs write files on their own at startup: nginx's docker diff immediately shows /run/nginx.pid and directories under /var/cache/nginx.

Clean up:

docker rm -f cow-demo

Tags

A tag is a label on the manifest. nginx:1.27.3, nginx:1.27, nginx:latest can point to the same image, or to different ones. They're just strings, and the repository owner can repoint them however they want (Pin base image versions).

Common conventions:

Tag	What it means
1.27.3	an exact patch release, rarely repointed, but it does happen (for example, on a rebuild against an updated base)
1.27	a minor series, moves to new patches
1	a major series, moves to new minor versions
latest	the default tag: used when no tag is given; what it points to is up to the maintainer
alpine, slim	variants on a minimal base, often paired with a version: 1.27.3-alpine

If you write docker pull nginx without a tag, Docker substitutes latest (docker image pull). The same happens with FROM nginx in a Dockerfile and with docker run nginx.

Why latest is a trap

The latest tag points to one specific image today. Tomorrow the maintainer ships a new release and repoints the tag to it. Your docker pull will fetch something different, even though the line in your Dockerfile hasn't changed.

What breaks:

• CI builds: it passed for a year, then one day it fails on a new major version of the base image.
• Production: servers pull the "latest" image at different times, one gets 1.27, another gets 1.28, behavior diverges.
• Security: the tag is mutable, and whoever can push to the repository can repoint it to a different image. You think the version is pinned, but it isn't.

Simple rule: in production, use an exact tag (1.27.3-alpine) or a digest. In dev an exact tag is also preferred, but 1.27 or 1 is acceptable.

Digest: an immutable reference

A tag can be repointed, a digest can't: it's a SHA256 hash of the manifest itself. Change the manifest, and the digest changes too. If an image is built for several platforms (amd64, arm64), the digest is computed from the shared manifest list, so it's the same on any machine.

Check the digest of an image you already have locally:

docker inspect nginx:1.27.3-alpine --format '{{index .RepoDigests 0}}'

nginx@sha256:814a8e88df978ade80e584cc5b333144b9372a8e3c98872d07137dbf3b44d0e4

docker pull prints the same digest in its Digest: line. You can substitute the full reference for a name-and-tag:

docker pull nginx@sha256:814a8e88df978ade80e584cc5b333144b9372a8e3c98872d07137dbf3b44d0e4

If that digest doesn't exist in the registry, Docker replies with manifest for nginx@sha256:... not found: manifest unknown. You can't swap the content under the same digest: different content gives a different hash.

In production, where you need reproducibility, you reference images by digest. Kubernetes manifests for critical services do the same. You can write both a tag and a digest at once (nginx:1.27.3-alpine@sha256:...): the tag stays for humans, and Docker pulls the image by digest.

An image you built or retagged yourself and never pushed has no digest from a registry. docker image ls --digests shows <none> for it.

Useful image commands

docker pull nginx:1.27.3-alpine         # download
docker images                           # list local images
docker image ls --digests               # with digest
docker history nginx:1.27.3-alpine      # which layers the image is built from
docker rmi nginx:1.27.3-alpine          # delete
docker image prune                      # remove dangling (unnamed) images

docker history shows every layer, the command that created it, and its size. The newest layer is on top:

docker history alpine:3.20.3

IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
511a44083d3a   2 years ago   CMD ["/bin/sh"]                                 0B        buildkit.dockerfile.v0
<missing>      2 years ago   ADD alpine-minirootfs-3.20.3-aarch64.tar.gz …   8.83MB    buildkit.dockerfile.v0

Only the top row has an ID, lower layers show <missing>, and that's normal for a downloaded image. Long commands are cut off with an ellipsis, the --no-trunc flag shows them in full. Layers sized 0B are metadata (CMD, ENV, EXPOSE), they hold no files. This command is a lifesaver when you can't tell where an image's extra size came from. The archive name depends on the architecture: on an Intel machine you'll see x86_64 instead of aarch64.

Be careful with docker image prune -a. It removes every image that currently has no container created from it, including images from your other projects. You'll have to download them again afterward.

Registry: where images live

A registry is a server that stores images and serves them over an HTTP API. There are many:

• Docker Hub (docker.io) is the default public registry, home to nginx, postgres, node and thousands of others. You can download without logging in, but the number of pulls is limited (Docker Hub usage and limits).
• GitLab Container Registry (registry.gitlab.com) is built into GitLab projects, a project's own CI usually pushes there.
• GitHub Container Registry (ghcr.io) is the GitHub equivalent.
• Cloud providers run their own registries too, convenient when your deploy target lives in the same cloud.

Logging in and working with a third-party registry:

docker login registry.gitlab.com
docker pull registry.gitlab.com/myorg/myrepo:1.0
docker push registry.gitlab.com/myorg/myrepo:1.0

The full image name is <registry>/<namespace>/<name>:<tag>. If <registry> is left out, docker.io is substituted. Docker Hub's official images use the library namespace, so nginx:1.27.3-alpine is actually docker.io/library/nginx:1.27.3-alpine: docker pull prints that string last.

Common mistakes

"I deleted the file, why didn't the image get smaller?" The file is deleted in the top layer, but it's still in a lower one. Check docker history: a heavy layer followed by a 0B layer where the file was deleted. Delete temporary files in the same RUN instruction where they were created (more on that in the Dockerfile lesson).

"I fixed a config inside the container, recreated it, the fix is gone." The fix lived in the old container's writable layer. docker rm deleted it along with the container.

"My machine and the server have the same tag, but behavior differs." The tag was repointed between two docker pull calls. Compare digests: docker image ls --digests on both machines.

docker image prune -a on your working machine. It wipes out images from every project that currently has no containers.

What you'll need for the tasks

• docker history <image> and docker history --no-trunc <image>
• docker diff <container>, the letters A, C, D
• docker exec <container> <command>
• docker pull <image>:<tag>, the Digest: line in the output
• docker image ls --digests and docker inspect <image> --format '{{index .RepoDigests 0}}'
• docker run -d --name … --memory … --cpus … <name>@sha256:<digest>

In the next lesson we'll build our own image with a Dockerfile.