Secrets management: Vault and sops

Why this matters

Almost every service needs secrets: a database password, a cloud token, a private key, a payment API key. It's tempting to put them straight into code or a config file, but that's a leak. Code lives in git, gets cloned, gets forked, and ends up in build logs.

You can remove a secret from git history (git filter-repo rewrites history), but that doesn't save you. Clones, forks, and CI caches already hold the old commits. So a secret that has made it into a repository even once is treated as leaked and gets changed.

Secrets management answers one question: where to keep secrets so that services have access, not people and repositories.

Concepts

Secret: a value that grants access: a password, a token, a key. Leaking a secret means leaking access.

External secrets store: a separate service that stores secrets and hands them out on request. Examples: HashiCorp Vault, OpenBao, AWS Secrets Manager, Yandex Lockbox.

File encryption: the secret lives in the repository, but encrypted. Tools: sops, Ansible Vault.

Rotation: regularly changing a secret. Least privilege: each service only has access to its own secrets.

Two approaches

They're often confused, even though they work very differently.

External store

The secret lives in a service, and the app fetches it at runtime using a token:

secret → [Vault] ← app requests it by token at startup

The secret never appears in code or in git, only the store's address and permission to request it. The store logs who read what and when, and can issue dynamic secrets: for example, a db password created for one specific service that expires on its own after a set time.

There's one thing worth knowing about Vault beyond the technical side. In August 2023 HashiCorp switched the license from the open MPL 2.0 to the Business Source License 1.1 (BSL), which restricts commercial use. In response, the community forked it as OpenBao. The fork is run by OpenSSF, a foundation inside the Linux Foundation. The architecture is the same, the API is compatible. If a company's legal team looks at licenses, the choice between Vault and OpenBao will come up.

File encryption

Ciphertext goes into git, and the decryption key is kept separately:

secrets.yaml (encrypted) → into git
decryption key → only on CI and with admins

This is handy for GitOps: everything lives in one repository, but secrets aren't in plaintext.

sops has a useful trait. In YAML and JSON it only encrypts the values and leaves the keys in the open. Here's what a file with a password and a token looks like:

db:
    user: ENC[AES256_GCM,data:kyo9,iv:...,type:str]
    password: ENC[AES256_GCM,data:QWwi71fXPSU9IlOAHBnNdw==,iv:...,type:str]
api_token: ENC[AES256_GCM,data:jJLqAipxGwNhWB4my0haZzHD,iv:...,type:str]
sops:
    age:
        - recipient: age1wqlp3sr8ej3m9cf8vnshsqmgm5vnahrnurrrj2jxr4q0tyxu8gdqd29p70
    ...

In a review you can see which field changed, even with the values hidden. The sops: block holds metadata: who the file can be decrypted for (one age recipient here) and a checksum.

sops takes keys from age, PGP, or a cloud KMS (AWS, GCP, Azure). The simplest option to start with is age: a key pair is created with one age-keygen command. The public key (age1...) can be shown to anyone: it's used for encrypting. The private key (AGE-SECRET-KEY-1...) is needed for decrypting, and it lives outside the repository.

Core commands:

age-keygen -o keys.txt

sops --encrypt --age age1... --in-place secrets.yaml

sops --decrypt secrets.yaml

Decryption looks for the private key in the file named by the SOPS_AGE_KEY_FILE variable, and if that isn't set, in a default location: ~/.config/sops/age/keys.txt on Linux, ~/Library/Application Support/sops/age/keys.txt on macOS. The keys.txt file from the example above gets moved there (or pointed to via SOPS_AGE_KEY_FILE), never into the repository. To avoid writing --age in every command, a .sops.yaml with rules goes into the repository root: which files get encrypted with which key. sops 3.8 looks for this file from the current folder upward, so you run commands from the repository root.

Ansible Vault works similarly, but by default encrypts the whole file at once, with a single password.

What to choose

• Dynamic infrastructure, many services, need to issue short-lived tokens: an external store (Vault, OpenBao, a cloud secrets manager).
• A small project, everything described in git, want to keep secrets next to the code: file encryption (sops).

They're often combined: static secrets via sops in git, dynamic and sensitive ones via Vault.

How the secret reaches the container

Wherever the secret is stored, it eventually has to be handed to the application. This is an easy place to create a leak out of nowhere.

A secret in a Dockerfile ends up in the image forever. ENV gets written into the image's config, and an ARG used by RUN shows up in the build history. Here's what docker history --no-trunc shows for an image built with ENV DB_PASSWORD=... and --build-arg BUILD_TOKEN=...:

RUN |1 BUILD_TOKEN=lab-only-build-arg /bin/sh -c echo "token len ${#BUILD_TOKEN}" > /tmp/t # buildkit
ARG BUILD_TOKEN
ENV DB_PASSWORD=lab-only-env-pass

Anyone who pulls the image from a registry can read this without any access to the database. If a secret is needed during the build (a token for a private package registry), it's passed via RUN --mount=type=secret: the file is mounted for the duration of one command and never shows up in the history.

At runtime, the secret is handed in from outside. An environment variable (-e, environment:) works, but anyone with Docker access can see it via docker inspect. A file is safer. Docker Compose has secrets: for this:

services:
  app:
    image: myapp:1.4.2
    secrets:
      - db_password
secrets:
  db_password:
    file: ./db_password.txt

Compose mounts the file read-only at /run/secrets/db_password, and the app reads it from there. docker inspect shows the path to the file, not the value. Many official images support this: for example, postgres has a POSTGRES_PASSWORD_FILE variable.

Rotation and access

Two principles make secrets safer.

Rotation. If a token leaks, a short lifetime limits the damage. External stores can issue dynamic secrets that expire automatically. A secret stored in an encrypted file gets changed by hand: a new value, a new commit.

Least privilege. Each service has access only to its own secrets, not to everything at once. That way breaking into one service doesn't open everything. In sops this means separate files under separate keys; in Vault, separate policies per service.

And one rule that cuts across everything: secrets never appear in the open in logs, backups, or image layers.

Common mistakes

• .gitignore after the commit. The file is already in history; .gitignore doesn't affect it. The secret gets changed, the file gets removed from the index.
• A password in a Dockerfile's ENV or ARG. Visible in docker history and docker image inspect to anyone who pulls the image.
• A secret in environment: instead of a file. Works, but the value can be read via docker inspect and from the process environment.
• A private age key next to the ciphertext. If keys.txt is committed into the same repository, the encryption protects nothing.
• Encrypted, but the plaintext version is already in the history. Encryption has to happen before the first commit. If you're too late, the secret counts as leaked.
• echo $DB_PASSWORD in a deploy script. The value stays in the CI logs.

What you'll need for the tasks

• Docker 20.10+ and Docker Compose v2: docker build, docker history --no-trunc, docker image inspect, docker inspect, docker compose up -d --build.
• Section "How the secret reaches the container": where ENV stays in the image, how secrets: works in Compose.
• Section "File encryption": age-keygen, sops --encrypt --age, sops --decrypt.
• git: git add, git commit, git log -p.

What to remember

Secrets can't be kept in code and git in the open. Two approaches: an external store (Vault, OpenBao, Lockbox) hands out a secret by token at runtime; file encryption (sops, Ansible Vault) puts ciphertext into git and keeps the key separately.

An external store is used for dynamic setups and many services. File encryption fits when everything is in git. They're often combined.

Rotation limits the damage from a leak; least privilege narrows its blast radius. Secrets don't end up in logs, backups, or image layers: they're handed to the container at startup, preferably as a file.

In the next lesson we'll cover what to do when something does break anyway: incident response and the postmortem.