Secrets: Ansible Vault

Playbooks and roles live in git, and database passwords, cloud tokens and keys want to tag along. In this lesson we'll cover how ansible-vault encrypts secrets so you can commit them safely, how to hand over the password by hand and in CI, and how to keep different environments' passwords apart.

The problem: secrets in the repository

Variables sit next to the playbook: in group_vars/, host_vars/, a role's vars/. If a database password is written there in plaintext, everyone with access to the repository sees it. It also ends up in the commit history, in forks and in mirrors.

Deleting the secret in the next commit isn't enough: the old commit stays in the history and with everyone who already cloned the repository. So a plaintext secret should never reach git at all. That's what Ansible Vault handles.

What ansible-vault does

ansible-vault encrypts data symmetrically (AES256) with a password. The input is plain YAML, the output is a blob safe to commit. Decryption needs the same password: when the playbook runs, Ansible gets it and decrypts the data in memory, never writing the plaintext to disk.

There are two ways to encrypt: a whole file, or a single value inside an ordinary vars file. Let's look at both.

Encrypting a whole file

# encrypt an existing file
ansible-vault encrypt group_vars/prod/secrets.yml

# create already encrypted
ansible-vault create group_vars/prod/secrets.yml

# edit (decrypts to a temp file, then back)
ansible-vault edit group_vars/prod/secrets.yml

# view without decrypting to disk
ansible-vault view group_vars/prod/secrets.yml

# change the password
ansible-vault rekey group_vars/prod/secrets.yml

Each command asks for the password (twice for create, encrypt and the new password in rekey). After encrypt the file starts with a header line, followed by the ciphertext:

$ANSIBLE_VAULT;1.1;AES256
36393839633462663066633764346563363635396634343636353930313833626463386638343237
...

Ansible recognizes such a file by its header. Once it's decrypted, Ansible reads plain YAML with variables inside.

Encrypting a single value

Encrypting a whole file isn't always convenient: non-secret variables also turn into ciphertext and disappear from diff. encrypt_string encrypts one value, and you drop the result right into an ordinary vars file:

ansible-vault encrypt_string 'hunter2' --name 'db_pass'

Many tutorials show it this way, but you shouldn't: the password ends up in shell history. It sits in plaintext in ~/.bash_history or ~/.zsh_history, and anyone who gets access to your home directory can find it. While the command is running, the password string is also visible in ps.

It's safer to read the value from stdin:

ansible-vault encrypt_string --stdin-name 'db_pass'

The command asks for the vault password, then prints Reading plaintext input from stdin. (ctrl-d to end input, twice if your content does not already have a newline). Paste the secret and press Ctrl-D twice, without pressing Enter. If you press Enter, the newline becomes part of the secret: instead of hunter2 Ansible decrypts hunter2\n, and the password won't match the database. Shell history only gets the command itself, never the secret.

Both options print a ready piece of YAML:

db_pass: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          66613833656366653534356265323463383663613664663561663464646336643938393061353964
          ...

You paste it into the vars file in place of the plaintext line. The other variables stay readable and visible in diff, only the db_pass value is encrypted.

How to hand over the password at run time

Without a password, a playbook with encrypted data fails:

[ERROR]: Attempting to decrypt but no vault secrets found.

There are three ways to pass the password:

# ask for the password interactively
ansible-playbook site.yml --ask-vault-pass

# take it from a file
ansible-playbook site.yml --vault-password-file ~/.vault_pass

# or through an environment variable
export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass
ansible-playbook site.yml

--vault-password-file accepts both a file with the password and an executable script that prints the password to stdout. This is how the password gets pulled from an external store or a password manager. The password file itself never goes into git: add it to .gitignore.

If the password is wrong, the error is different:

[ERROR]: Decryption failed (no vault secrets were found that could decrypt).

Several environments: vault-id

If each environment (dev, prod) has its own password, vault-id tells them apart. It's a label plus a password source in the form label@file:

ansible-vault encrypt --vault-id dev@~/.vault_dev group_vars/dev/secrets.yml

The label goes into the file's header, and the format becomes 1.2:

$ANSIBLE_VAULT;1.2;AES256;dev

At run time you pass as many --vault-id flags as there are environments involved:

ansible-playbook site.yml \
  --vault-id dev@~/.vault_dev \
  --vault-id prod@~/.vault_prod

The label in the header tells Ansible which password to try first. But by default it also tries the other passwords it was given. So an "encrypted with the wrong password" mistake is easy to miss while all the passwords travel together. It surfaces once a run gets only one of them, say a prod deploy with a single --vault-id prod@....

Vault in CI

A pipeline has no interactive input, and you can't put the password in the repository. The scheme looks like this:

1. The password sits in a CI/CD variable of the project. In GitLab it's marked protected (available only to pipelines on protected branches and tags) and masked (the value is hidden in job logs).
2. The job writes the variable to a temporary file and runs the playbook with --vault-password-file.
3. Developers commit the encrypted files. The prod password doesn't have to be handed to them at all: prod secrets are decrypted only in CI.

# .gitlab-ci.yml (fragment)
deploy:
  script:
    - install -m 600 /dev/null /tmp/vault_pass        # create the file with 600 permissions right away
    - echo "$ANSIBLE_VAULT_PASS" > /tmp/vault_pass
    - ansible-playbook site.yml --vault-password-file /tmp/vault_pass

Ansible trims the trailing newline from the password file, so echo is safe here. GitLab also has File-type variables: the runner writes the value to a temporary file itself, and the variable holds the path to it. Then the first two lines of the script aren't needed.

When Vault isn't the best choice

Vault works well while there are few secrets and they rarely change. Once there are many secrets that need regular rotation and different access for different teams, encrypted files in git become awkward: every secret change is a commit, and access is split only by the vault password.

At that point secrets live in an external store (HashiCorp Vault, a cloud secrets manager) and get fetched at run time through a lookup or a module. The repository has no secrets at all, and the pipeline reaches the store with a short-lived token or through OIDC.

Common mistakes

• Secret as a command-line argument. encrypt_string 'password' leaves the password in shell history. Pass the value through --stdin-name.
• Enter before Ctrl-D. The newline ends up inside the secret, and the decrypted value no longer matches the password. You can check like this: ansible <host> -m debug -a var=db_pass --vault-password-file <file>. If the value ends with \n, encrypt it again.
• Password file in the repository. A .vault_pass next to the playbook without .gitignore defeats the encryption entirely.
• Secret already leaked into git. Encrypting it in the next commit isn't enough: the old value stays in the history. Rotate the secret wherever it is used (in the database, in the cloud), and put only the new one into vault.
• Encrypted with the wrong password. Running with every --vault-id at once hides this, only a run with a single password fails. Check the label in the file's first line.

What you'll need for the tasks

Ansible is already installed inside the lab environment, commands run through bash setup/run.sh.

• ansible-vault encrypt_string --stdin-name <name> and --vault-password-file <file>;
• ansible-vault view, encrypt, decrypt, rekey and their --help;
• --vault-id <label>@<file> for ansible-vault and ansible-playbook;
• the header $ANSIBLE_VAULT;... in the first line of an encrypted file;
• ansible <host> -m debug -a var=<name> to see the decrypted value.

Next up is a short quiz covering the whole ansible-basics track.