Image registries and tags

A built image needs somewhere to live so other machines, CI, and servers can pull it. This lesson covers how a full image name is put together, what docker push and docker pull do, how a tag differs from a digest, and how to log in to a private registry.

Where images live

The storage for images is called a registry. Docker Hub (docker.io) is the public default registry. GitLab has its own registry (registry.gitlab.com), and so do clouds and companies that run one in-house. Most of them are private: no login, no pull or push.

For Docker, a registry is just an HTTP server with a specific API. The client asks it for an image's manifest and layers, and uploads them one by one on push. In this lesson's labs, you'll bring up one of these servers on your own laptop.

Full image name

A full image name has three parts: registry address, repository path, and tag.

registry.gitlab.com/team/app:1.4.2

Here registry.gitlab.com is the registry address, team/app is the repository path, and 1.4.2 is the tag. The address can include a port: 127.0.0.1:5055/team/app:1.0.

If there's no address at the start, Docker fills in Docker Hub. For official images it also adds the path library/. You can see this in the last line of docker pull output:

docker pull busybox:1.36.1

1.36.1: Pulling from library/busybox
Digest: sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662
Status: Image is up to date for busybox:1.36.1
docker.io/library/busybox:1.36.1

So busybox:1.36.1 and docker.io/library/busybox:1.36.1 are the same name.

push and pull

Push your image to a registry and pull it back:

docker build -t registry.gitlab.com/team/app:1.4.2 .
docker push registry.gitlab.com/team/app:1.4.2
docker pull registry.gitlab.com/team/app:1.4.2

push takes the registry address from the image name; there's no separate flag for "where to". So the name at build time (-t) already has to include the address. For an image you've already built, you can give it a second name with docker tag:

docker tag app:1.4.2 registry.gitlab.com/team/app:1.4.2

For an image without an address in its name, Docker tries to send it to Docker Hub. If you don't have that repository there, the push ends in an error:

The push refers to repository [docker.io/izi-lab-fc/app]
...
denied: requested access to the resource is denied

A registry stores an image by layers, and each layer is addressed by its own digest. Before uploading, the client asks the registry whether it already has a layer with that digest, and skips sending it if it does. Here's a second push of an image where only the top layer changed:

The push refers to repository [127.0.0.1:5057/team/app]
2550eb7fa25b: Preparing
651d9022c234: Preparing
651d9022c234: Layer already exists
2550eb7fa25b: Pushed
1.0: digest: sha256:35e05553a7b98e34d272ab4308ecd6f0d8861daeab0a794eb4e14addefb2508d size: 735

The base alpine layer (651d9022c234) was already in the registry, only the new one went up. pull works the same way: Docker doesn't download layers that are already present locally, and prints Already exists.

Tag and digest

A tag (:1.4.2, :stable) is a mutable label. It's just a name the registry links to a specific image, and anyone with push access can repoint it to a different image. Today app:stable points to one image, tomorrow, after a new push, to another. The tag Docker fills in when you don't specify one behaves the same way (covered in the lesson about images).

A version-numbered tag can be repointed too. In this test, the registry received two pushes of team/app:1.0 with different content:

1.0: digest: sha256:99ffdfdde7a3fe3876d57cbba908a2e7a4ba46d01742d5b1313b94eedaff6791 size: 735
...
1.0: digest: sha256:35e05553a7b98e34d272ab4308ecd6f0d8861daeab0a794eb4e14addefb2508d size: 735

After the second push, docker pull ...:1.0 returns the new image. The old one is still there, but you can only get it back by digest.

A digest (@sha256:...) is the SHA256 hash of the image's manifest. The manifest lists the digest of every layer and the config, so changing even one byte of the image and keeping the same digest is impossible. One digest always means one and the same image:

docker pull busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662

An image's digest is printed by both docker push and docker pull (the digest: or Digest: line), and for local images you can see it with docker image ls --digests.

A rule for production: a deployment references a specific version tag (:1.4.2), and wherever you need a guarantee that it's exactly the same image, a digest. Floating tags like :stable are fine for people, but not for deployment manifests.

Private registries and login

A private registry requires a login before push and pull:

docker login registry.gitlab.com

Docker asks for a username and password (or access token) and saves them. Exactly where depends on your setup: Docker Desktop stores them in the OS's own store (macOS Keychain, Windows Credential Manager), and without it they sit in ~/.docker/config.json in base64, which is effectively plain text (docker login).

A saved login is tied to the registry address exactly as it was passed to docker login. On push and pull, Docker takes the address from the image name and looks up a login under that key. Logged in to gitlab.example.com, but the image is named registry.gitlab.example.com/team/app:1.4.2? No login will be found, and push ends like this:

no basic auth credentials

This means the client didn't find a login for the address in the image name, and didn't even send a password. The password isn't the issue here: compare the address in docker login with the start of the image name.

In scripts and CI, the password is passed through --password-stdin, not the -p flag: that way it doesn't end up in shell history or the process list. Docker itself warns about -p:

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

In GitLab CI, you log in with built-in variables, not a password in the repository. CI_REGISTRY holds the project's registry address, CI_REGISTRY_USER and CI_REGISTRY_PASSWORD are issued for the duration of the job, and CI_REGISTRY_IMAGE is the project's full image name without a tag (predefined variables, authenticating with the container registry):

echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
docker build -t "$CI_REGISTRY_IMAGE:1.4.2" .
docker push "$CI_REGISTRY_IMAGE:1.4.2"

Cleaning up a registry

Every CI build adds a new image to the registry, and space runs out fast. That's why you set up a cleanup policy: for example, delete tags older than 90 days but always keep the last 10. In GitLab this is the cleanup policy in the project settings (docs).

Deleting a tag doesn't free space right away. Layers can be used by other images, so the registry removes them in a separate garbage collection pass, once nothing references them anymore (garbage collection in distribution).

Common mistakes

• You built an image without a registry address in its name and ran push. Docker went to Docker Hub and replied denied: requested access to the resource is denied. You need a name with an address: docker tag or rebuild with -t.
• You logged in to one address, but the image name has a different one. The login is saved, but push prints no basic auth credentials.
• A password sits in .gitlab-ci.yml or a script in plain text. It ends up in git and in job logs. Take the password from CI variables.
• You repointed a version tag with a new push and expect servers that already pulled the image to pick up the change. Docker doesn't re-download an image if the tag already exists locally, unless you run docker pull explicitly.
• A registry over plain HTTP without TLS at an address other than 127.0.0.0/8. Docker doesn't accept such registries by default; they need to be explicitly allowed in the daemon's insecure-registries (dockerd, insecure registries). In production you use TLS instead.

What you'll need for the tasks

• Full image name: <registry address>/<path>:<tag>, for example 127.0.0.1:5055/team/app:1.0.
• docker build -t <name> <folder>, docker tag, docker push, docker pull, docker rmi, docker run --rm.
• The registry API through curl: curl -s http://<address>/v2/_catalog (list of repositories) and curl -s http://<address>/v2/<path>/tags/list (tags).
• echo "$PASSWORD" | docker login <address> -u <user> --password-stdin and the rule: the address in the login matches the address in the image name.