ServiceAccount and the token in a Pod

What identity code inside a Pod uses to talk to the cluster API, where the token comes from, and why you should turn off automount where it isn't needed.

Why this matters

Code in a Pod sometimes calls kube-apiserver on its own: an operator watches objects, a CI agent launches a Job, an app reads its own ConfigMaps. The apiserver has to figure out who's asking and decide what they're allowed to do. For code in a Pod, the answer to "who" comes from the ServiceAccount.

The flip side: a token sits in every Pod by default, even when the app never needs the API. Compromise that Pod, and the attacker gets ready-made credentials for the cluster.

What a ServiceAccount is

A ServiceAccount (SA) is the identity code inside a Pod uses to talk to kube-apiserver. Every SA has its own name in the form system:serviceaccount:<namespace>:<sa-name>. The apiserver plugs this name in as the subject when it checks RBAC: what this subject is allowed to do per the Role and RoleBinding.

A ServiceAccount lives inside a namespace, like a ConfigMap or Secret. Kubernetes creates an SA named default in every namespace on its own. If a Pod's spec doesn't set an SA, the apiserver substitutes default from the same namespace when creating the Pod (docs).

This hides a trap. Every Pod in a namespace without its own SA runs as the same default. Grant it permissions for one app, and every one of those Pods gets them. A ClusterRoleBinding on default spreads the permissions even wider.

How to create an SA

One command:

kubectl create serviceaccount app-sa -n demo

Under the hood this creates an object:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: demo

An SA by itself grants no permissions. It's just an identity. To get permissions you need a Role and RoleBinding (or a ClusterRole + ClusterRoleBinding for cluster-wide access). The next lesson covers RBAC.

A RoleBinding can reference an SA that doesn't exist yet. There's no error: the permissions just start working once an SA with that name shows up.

How an SA gets into a Pod

In the Pod's spec:

spec:
  serviceAccountName: app-sa
  containers:
    - ...

If you don't set it, default gets substituted. The field is only set when the Pod is created: you can't change it on an existing Pod, you have to recreate it. For a Deployment, you change the SA in the Pod template (spec.template.spec.serviceAccountName), and the Deployment replaces the Pods with new ones on its own. There's also a short command for this, kubectl set serviceaccount.

Where the token inside a Pod comes from

When the Pod starts, kubelet mounts a projected volume into the container at /var/run/secrets/kubernetes.io/serviceaccount/. It holds three files:

File	What's inside
token	the SA's JWT token (short-lived, renewed automatically)
ca.crt	kube-apiserver's certificate, for TLS verification
namespace	the name of the namespace the Pod lives in

In the Pod's spec, this volume shows up under a name like kube-api-access-xxxxx.

Before 1.24, every SA automatically got its own Secret with a token that never expired, and kubelet just mounted it. Now kubelet requests a token through the TokenRequest API. The token is bound to the Pod and is issued for about an hour by default (3607 seconds). Kubelet handles renewal itself: it reissues the token once 80% of its lifetime has passed, or once the token turns a day old, whichever comes first (docs). Old Secret-based tokens are still accepted, but new ones aren't created automatically anymore, and the cluster has cleaned up unused legacy tokens on its own since 1.29.

One caveat about the lifetime. On kubeadm clusters (kind runs on it too), the apiserver has the flag --service-account-extend-token-expiration enabled by default: a token that kubelet requested is issued for a year, and the hour becomes the point after which the apiserver writes a warning to the audit log (flag reference). This is a transitional measure for old clients that don't re-read the token file. So you might see an exp field dated a year out.

You can look at the token from inside the Pod like this:

kubectl exec probe -n demo -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/token

You'll get a long string made of three parts separated by dots. That's a JWT: header.payload.signature. The payload is base64url-encoded:

kubectl exec probe -n demo -- sh -c \
  'cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d. -f2 | base64 -d 2>/dev/null'

Base64url drops the trailing = characters, so base64 -d may cut off the last few bytes. The start of the JSON reads fine. Inside you'll find sub, aud, exp, and a kubernetes.io block with a reference to the SA: serviceaccount.name=app-sa, pod.name=probe.

How the token is used

Any HTTP client in the Pod can call the API. For example, curl in an image that has it (busybox doesn't):

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
NS=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)

curl --cacert $CA -H "Authorization: Bearer $TOKEN" \
  https://kubernetes.default.svc/api/v1/namespaces/$NS/pods

The apiserver checks the token's signature, sees the SA as the subject, and checks RBAC. If the SA lacks the list pods permission in the namespace, the response is a 403 with text like pods is forbidden: User "system:serviceaccount:demo:app-sa" cannot list resource "pods". That name in the error message tells you right away which SA the Pod is running as. If there's no token at all, the request goes out with no identity: the response is 401 or 403 with the user system:anonymous, depending on whether anonymous requests are allowed in the cluster.

You can also check an SA's permissions from outside, without entering the Pod:

kubectl auth can-i list pods -n demo \
  --as=system:serviceaccount:demo:app-sa

The answer is yes or no.

How to turn off the mount

Sometimes a Pod doesn't need the token: the app never calls the API. Then you turn off automount, so a compromised Pod doesn't hand over easy cluster access:

spec:
  automountServiceAccountToken: false
  serviceAccountName: app-sa

Or at the level of the SA itself, to turn it off for every Pod using that SA:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
automountServiceAccountToken: false

The field in the Pod's spec takes priority over the field on the SA (docs). So you can turn off the token for everyone on the SA, and bring it back for one Pod with automountServiceAccountToken: true in its spec.

Both fields are read when the Pod is created. A Pod that's already running keeps its mounted token, only new Pods see the change. Automount is on by default. That's a historical holdover, and audits often require turning it off where it isn't needed.

See it for yourself

kubectl create namespace demo
kubectl create serviceaccount app-sa -n demo

kubectl run probe -n demo --image=busybox:1.36.1 \
  --overrides='{"spec":{"serviceAccountName":"app-sa"}}' \
  -- sleep 3600

kubectl exec probe -n demo -- \
  ls -la /var/run/secrets/kubernetes.io/serviceaccount/

kubectl exec probe -n demo -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/namespace

A Pod sees itself through the API using the token in the mounted volume. Every operator and in-cluster client is built on this idea.

Common mistakes

• Granting permissions to SA default because a Pod runs as it. Every Pod in the namespace without its own SA gets those permissions too. The better fix is to give the app its own SA and set it in the Pod template.
• Creating an SA and expecting a Pod to start using it on its own. Until serviceAccountName is set, the Pod runs as default.
• Trying to change serviceAccountName on a live Pod via kubectl edit. The API refuses, the Pod has to be recreated, and for a Deployment you edit the template instead.
• Turning off automount on an SA that several apps share. The token disappears for all of them, including the ones that need the API. This doesn't break anything right away, only on the next Pod restart.
• Not reading the username in the 403 error's text. system:serviceaccount:... means the token is there and this SA just lacks permissions. system:anonymous means there's no token, and fixing RBAC for the SA won't help.

What you'll need for the tasks

• kubectl create serviceaccount, kubectl get sa.
• serviceAccountName in a Deployment's Pod template, and kubectl set serviceaccount.
• automountServiceAccountToken in the Pod spec and on the SA, and the priority rule.
• kubectl logs and the 403 error's text, to find out which SA a Pod runs as.
• kubectl auth can-i ... --as=system:serviceaccount:<ns>:<sa>.
• kubectl get pod <name> -o yaml, to find the kube-api-access-* volume and volumeMounts.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your machine, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-serviceaccount-token     # set up the starting state
./run.sh verify k8s-serviceaccount-token    # check the current step

In short

• A ServiceAccount is a Pod's identity in the apiserver's eyes, named system:serviceaccount:<ns>:<sa>.
• Without spec.serviceAccountName, a Pod gets default from its own namespace.
• Kubelet mounts token, ca.crt, and namespace into /var/run/secrets/kubernetes.io/serviceaccount/.
• Since 1.24, the TokenRequest API issues the token: it's bound to the Pod and gets reissued automatically.
• An SA by itself grants no permissions, those come from a Role and RoleBinding.
• automountServiceAccountToken: false removes the token from a Pod that doesn't need the API. The field on the Pod outranks the field on the SA.