Secret and why base64 is not encryption

What a Secret is

Secret - a Kubernetes object for sensitive data: passwords, tokens, keys. In structure it's close to ConfigMap. Both store key/value pairs in the cluster API (backed by etcd), and both connect to a Pod as env or as files. Values in a Secret's data are written in base64, and a Secret has a type field.

The rest of the difference is in how Kubernetes itself treats a Secret. Read permissions on a Secret in RBAC are granted separately from ConfigMap. kubelet only gets the Secrets needed by Pods on its own node, and a secret volume puts files in tmpfs, that is, in memory, not on the node's disk. The data in a single Secret is capped at 1 MiB.

The main confusion for newcomers: base64 is not encryption. It's an encoding, so binary data can go into YAML. Anyone who gets a Secret via kubectl get -o yaml decodes it in a second with base64 -d. Inside the cluster, a Secret is protected by RBAC (who can read it) and etcd encryption (what form it takes on disk).

etcd encryption (encryption at rest) is off by default. The cluster admin turns it on in the kube-apiserver settings. In a learning kind cluster it's absent: the password from a Secret sits in etcd in plain text, and you can see that by reading the key directly from etcd.

So why have a Secret at all, if it's encoded rather than encrypted? Three reasons. kubectl describe pod shows only a reference for env from a Secret, never the value. etcd encryption can be turned on for all Secrets at once, without touching any app. External tools (Vault, External Secrets, Sealed Secrets) can create and update plain Secrets, and the app never knows they're involved.

Minimal YAML

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
data:
  username: YWRtaW4=
  password: czNjcmV0

YWRtaW4= is admin in base64. Encode it like this:

echo -n admin | base64

-n is required, otherwise echo adds a newline, and the Secret ends up with YWRtaW4K, that is, admin with a \n at the end. This is a very common trap. printf %s admin | base64 gives the same result as echo -n.

If you put something other than base64 into data, the apiserver refuses to create it: illegal base64 data at input byte ....

You don't have to encode by hand. The stringData field accepts plain-text values, the apiserver encodes them into base64 itself and moves them into data:

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
stringData:
  username: admin
  password: s3cret

On read, stringData is already gone, only data remains.

The type field

• Opaque. Arbitrary keys, the usual case. It's the default when type isn't set.
• kubernetes.io/dockerconfigjson. For image pull secrets, needs a key .dockerconfigjson with JSON in the ~/.docker/config.json format.
• kubernetes.io/tls. For TLS certificates, needs keys tls.crt and tls.key.
• kubernetes.io/service-account-token. A long-lived ServiceAccount token. Since Kubernetes 1.24, such Secrets aren't created automatically for a ServiceAccount; Pods get a short-lived token via a projected volume instead.

Each special type has its own set of required keys, and the apiserver checks them. For example, a Secret of type kubernetes.io/tls without tls.key won't be created: data[tls.key]: Required value.

Connecting to a Pod

Nearly identical to ConfigMap, just secretKeyRef instead of configMapKeyRef:

env:
  - name: DB_USER
    valueFrom:
      secretKeyRef:
        name: db-creds
        key: username

And a volume:

volumes:
  - name: creds
    secret:
      secretName: db-creds
      defaultMode: 0400

Note: ConfigMap's field is called name, Secret's is secretName. envFrom with secretRef also exists and works the same way as with ConfigMap.

defaultMode: 0400 sets permissions to "read for the owner", nothing for anyone else. The number is written with a leading zero, that's octal notation. In the API it's stored as decimal, kubectl get pod -o yaml will show 256. The files in the volume are owned by root. If the process in the container doesn't run as root, 0400 locks the file for it too, and you'll need fsGroup in the Pod's securityContext.

Env and a file behave differently when a Secret changes. Env is read once, at container start; a file in a volume is refreshed by kubelet on its own, with a delay of around a minute (except for files mounted via subPath). Same as with ConfigMap.

automountServiceAccountToken

By default, Kubernetes mounts a ServiceAccount token, the access key to the cluster API, into every Pod. It lives at /var/run/secrets/kubernetes.io/serviceaccount/token. If the app doesn't call the API, the token is a spare secret sitting inside the Pod for nothing. Turn it off in the Pod's spec:

spec:
  automountServiceAccountToken: false

Fewer access rights inside the Pod, less to steal if the app gets compromised.

Related commands

kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password=s3cret
kubectl create secret tls my-tls --cert=cert.pem --key=key.pem
kubectl get secret db-creds -o yaml
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
kubectl describe secret db-creds
kubectl explain secret.type

kubectl describe secret only shows the size of values in bytes, for example password: 6 bytes. Handy for spotting a stray \n without revealing the password itself.

--from-literal encodes the value itself, with no extra newline. --from-file takes the whole file, including a trailing \n if one's there.

Common mistakes

• Committing a Secret to git as a plain YAML file. That's a leak: anyone can decode base64. Put SealedSecrets, External Secrets references, or SOPS-encrypted files into git instead.
• Thinking a Secret is encrypted. It's encoded. If the data matters, turn on etcd encryption and restrict who can read the Secret via RBAC.
• Encoding with echo without -n, which adds a \n to the value.
• Changing a Secret without the env in the Pod updating (same as with ConfigMap). You need to recreate the Pod; for a Deployment that's kubectl rollout restart. Or read the value from a file in a volume instead.
• defaultMode: 0400 when the process doesn't run as root: the app gets Permission denied on its own password.

What you'll need for the tasks

• Lab 1 (labs/01-db-secret): kubectl create secret generic --from-literal, reading via jsonpath and base64 -d, secretKeyRef, a secret volume with defaultMode, automountServiceAccountToken: false.
• Lab 2 (labs/02-api-auth): comparing the value in a Secret with the value in the process env (kubectl exec ... printenv), data versus stringData, recreating a Pod via kubectl rollout restart.
• Commands: kubectl get secret -o yaml, kubectl describe pod, kubectl logs, kubectl exec.

Homework

Practice the theory on your own cluster: 4 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-secret     # set up the starting state
./run.sh verify k8s-secret    # check the current step

In short

• A Secret stores key/value pairs in base64 and has a type field; its RBAC permissions are separate, and in a volume it lives in the node's memory.
• base64 is an encoding, not encryption: base64 -d exposes the value in a second.
• What actually protects secrets are RBAC, etcd encryption, and external stores like Vault.
• echo -n is required when encoding by hand. --from-literal or stringData are simpler.
• automountServiceAccountToken: false removes a spare API access token from the Pod.