Secrets in containers

Why

A container almost always needs a secret: a database password, an API token, a key. The simplest ways to pass it are exactly the ones that leak. The password ends up in the image, and everyone with registry access can pull that image.

In this lesson we'll look at where secrets leak, how to hand them to a container at startup, and how to pass a secret into a build without it ending up in the layers.

Concepts

Secret — any value that grants access: a password, a token, a private key. Anyone else who sees it gets that access too.

Image layer — an immutable filesystem snapshot after one Dockerfile instruction. You can delete a file in the next layer, but it stays in the previous one. More detail in the lesson on images.

Image history — the list of instructions an image was built from. docker history shows it, and it travels with the image to the registry.

Where secrets leak

A secret in ENV. A password written into Dockerfile through ENV ends up in the image config. Two commands show it right away:

docker history --no-trunc myimage:1.0
docker image inspect -f '{{json .Config.Env}}' myimage:1.0

["PATH=/usr/local/sbin:...","DB_PASSWORD=env-lab-only-password"]

A secret in a build arg. ARG looks safer: the value isn't in Dockerfile, it's passed in from outside through --build-arg. But the value still lands in the history. Every RUN instruction after the ARG declaration keeps all the arguments in scope in the history, even ones the command doesn't use:

RUN |1 API_TOKEN=arg-lab-only-token /bin/sh -c echo hello > /hello.txt # buildkit

A secret in a file that got deleted later. COPY id_rsa followed by RUN rm id_rsa doesn't help. The file sits in the layer from COPY, and anyone who unpacks the image with docker save can get it out.

Rule: a secret must never reach the layers or history of an image. Images get distributed, so there can be no secrets in them.

Runtime secrets: variables and files

Hand a secret to a container at startup, not during the build. The simple way: an environment variable.

docker run -e DB_PASSWORD="$DB_PASSWORD" myapp:1.0

The secret is no longer in the image, it only exists in the running container. But the variable is visible to everyone with Docker access:

docker inspect -f '{{json .Config.Env}}' <container>

["DB_PASSWORD=run-lab-only",...]

On top of that, every child process inherits the variables, and they sit in /proc/<pid>/environ. Any debug dump of the environment, or a crashed process that logs its env, carries the secret away with it.

Handing the secret over as a file is more reliable. Compose has secrets: for that:

services:
  app:
    image: myapp:1.0
    secrets:
      - db_password
secrets:
  db_password:
    file: ./db_password.txt

Compose places the file in the container at the path /run/secrets/db_password. The app reads the password from the file, not from a variable. docker inspect shows no content, only the path to the source file on the host:

bind /home/user/project/db_password.txt -> /run/secrets/db_password

Without Swarm, a secret like this is just a bind mount of a single file. So the db_password.txt file on the host itself needs protecting: 600 permissions and a line in .gitignore.

If a secret leaks

A secret that has ever landed in an image, in Git, or in a log counts as leaked. Deleting the image from the registry isn't enough: someone may have pulled it earlier. The value has to change.

Rotating a secret is done once the old value is no longer accepted anywhere, and everyone who uses the secret is on the new one. For a redis password, that's redis itself and every service that connects to it. Change it on only one side, and the service stops authenticating.

Build-time secrets: BuildKit

Sometimes a secret is needed specifically during the build, say, a token to download a private dependency. ARG doesn't fit here, and COPY even less so. For cases like this there are BuildKit secrets: the file is mounted only for the duration of one RUN instruction and never reaches the layers.

# syntax=docker/dockerfile:1
FROM node:22.11.0-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npmtoken \
    NPM_TOKEN=$(cat /run/secrets/npmtoken) npm ci

docker build --secret id=npmtoken,src=./npm_token -t myapp:1.0 .

Inside this RUN, the token sits at /run/secrets/npmtoken. In the next instruction the file is already gone, the history shows only the command itself, and the docker save dump has no token value in it.

BuildKit is on by default in Docker Desktop and in Docker Engine 23.0 and newer. On Linux with an older Engine and no BuildKit, the build fails with the --mount option requires BuildKit. In that case, build with the DOCKER_BUILDKIT=1 variable.

Common mistakes

Using ARG for a token "because it's not in Dockerfile." The value isn't in the file, but it is in docker history. And in every RUN after the ARG, not just the one that uses the token.

Deleting the secret in the next layer. RUN rm secret.txt creates a new layer, the old layer with the file stays in the image. Same with RUN ... && rm split across instructions.

Password in environment: in docker-compose.yml, file in Git. The image is clean, but the secret ended up in the repository. Keep values in a file outside Git, and reference it in compose through secrets: or env_file:.

The app reads only a variable. The password moved to /run/secrets/..., but the code still expects DB_PASSWORD and fails. Many official images understand a _FILE-suffixed variant (POSTGRES_PASSWORD_FILE=/run/secrets/db_password). Teach your own app to read the file.

--mount=type=secret without BuildKit. The classic builder doesn't know the --mount flag and fails. Check that the build is going through BuildKit.

Where to store secrets centrally

A file on the server and secrets: in compose work for a single host. Once there are many servers and services, secrets live in an external store (HashiCorp Vault, cloud secret managers) or get encrypted right in the repository through sops. The container still gets the value handed to it at startup. That's a topic for the ops track.

What you'll need for the tasks

• docker history --no-trunc <image> and docker image inspect <image>
• docker inspect <container>, the Config.Env and Mounts fields
• secrets: in docker-compose.yml and the path /run/secrets/<name>
• docker compose up -d --build, docker compose logs, docker compose ps
• docker compose restart <service>
• RUN --mount=type=secret,id=... and docker build --secret id=...,src=...
• docker save <image> | grep -a -c <string>, to check whether the string is in the layers