Forbidden for a ServiceAccount

An app inside the cluster calls the Kubernetes API and gets rejected:

Error from server (Forbidden): configmaps is forbidden: User "system:serviceaccount:demo:app-sa" cannot list resource "configmaps" in API group "" in the namespace "demo"

From the outside everything looks set up: the ServiceAccount exists, the Role exists, the RoleBinding exists. And the request still doesn't go through. In this lesson you'll learn to read this error word by word and check permissions with a command, instead of guessing.

Concepts

ServiceAccount (SA): an account for processes inside the cluster. A Pod runs as the SA named in the field spec.serviceAccountName. If the field isn't set, the Pod gets the default SA of its namespace.

In errors and permission checks, an SA is written as a subject in the form system:serviceaccount:<namespace>:<name>. The namespace is part of the name: SA app-sa in demo and SA app-sa in default are two different subjects for the apiserver.

A Role describes permissions (what you can do), a RoleBinding grants them (to whom). A 403 Forbidden response means the apiserver recognized the client, but no rule allows this action. Kubernetes permissions are allow-only; there are no deny rules: whatever isn't in the rules isn't allowed (docs).

Four reasons for Forbidden

When an SA gets a 403 from kube-apiserver, the cause is one of four:

1. The Pod doesn't specify its own SA, and uses default, which has no permissions.
2. The SA exists, but there's no RoleBinding or ClusterRoleBinding.
3. Everything exists, but the Role is missing the needed verb or resource.
4. The subject in the RoleBinding is set with a different namespace.

The third one is tricky because everything looks in place. Someone wrote a Role for pods and forgot about configmaps. Or granted get but not list. The symptom looks the same.

The main tool: kubectl auth can-i

The command answers the question "can this subject perform this action". Without it, you'd have to debug RBAC by trial and error.

kubectl auth can-i list configmaps \
  --as=system:serviceaccount:demo:app-sa -n demo

no

The answer is yes or no. The --as flag turns on impersonation: kubectl sends the apiserver a SelfSubjectAccessReview request as the given subject, and the apiserver checks it with the same rules as a request from the real client. To use --as, your account needs the impersonate permission. A cluster administrator, including in kind, has it.

The subject format matters: system:serviceaccount:<namespace>:<sa-name>. Without the prefix, kubectl reads the string as a username, not an SA, and the answer will be about a different subject.

The -n flag matters too. Permissions on namespaced resources depend on the namespace. If you skip the flag, kubectl takes the namespace from the current context, and that might not be where the SA lives. The namespace in --as and in -n are different things: the first says whose SA it is, the second says where the action is checked.

The full picture for a subject comes from one command:

kubectl auth can-i --list \
  --as=system:serviceaccount:demo:app-sa -n demo

Resources                                       Non-Resource URLs   Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                  []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
pods                                            []                  []               [get list]
                                                [/api]              []               [get]
...
configmaps                                      []                  []               [get]

Here you can see that configmaps only has get, not list.

The table shows every allowed action, including the built-in ones that any authenticated client has (selfsubjectreviews, /api, and other paths without a resource). It's worth checking before making changes, to see exactly what's missing.

Decoding the error

The error text is structured, and it has everything you need to fix it:

• subject: system:serviceaccount:demo:app-sa
• verb: list
• resource: configmaps
• apiGroup: "", that is, the core API, hence no prefix
• namespace: demo

Read from the first word. If the subject isn't the SA the app is supposed to run as (for example, ...:demo:default), the Role has nothing to do with it: the Pod is running under the wrong SA.

If the subject is correct, the Role inside namespace demo needs a rule with apiGroups: [""], resources: ["configmaps"], and verbs: ["list"]. get and watch are often added right along with list: clients that watch objects (controllers, the informer in client-go) do a list first, then a watch (docs).

A wildcard resources: ["*"] would also clear the error, but it gives the SA permissions on everything in the namespace, including Secrets. Don't do that.

Where to look at existing rules

# All Roles in the namespace
kubectl get role -n demo

# Full YAML of a specific Role
kubectl get role app-reader -n demo -o yaml

# Who the Role is granted to: columns ROLE, USERS, GROUPS, SERVICEACCOUNTS
kubectl get rolebinding -n demo -o wide

In the output of get rolebinding -o wide, the SA is written as <namespace>/<name>. This is where you spot the fourth cause: default/app-sa instead of demo/app-sa.

rules is an array. Each element has three required fields: apiGroups, resources, verbs. They work as a Cartesian product: every verb is allowed on every resource from every apiGroup within one rule.

Example Role with two rules:

rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]

With this Role you can do get pods, list pods, and get configmaps. You can't: list configmaps (no verb), delete pods (no verb), get secrets (no resource).

Role vs ClusterRole

Four objects that are easy to mix up:

Object	Scope
Role	permissions inside one namespace
ClusterRole	permissions at the cluster level, or a template for multiple namespaces
RoleBinding	grants permissions (from a Role or ClusterRole) inside one namespace
ClusterRoleBinding	grants permissions (only from a ClusterRole) across the whole cluster

RoleBinding plus ClusterRole is a working and common combination: a shared permission template gets applied precisely in one namespace.

How fixes are made

Add the missing verb to the existing Role:

kubectl edit role app-reader -n demo

or patch it without an editor. For the Role from the example above, the rule for configmaps is second in the array, that is, at index 1:

kubectl patch role app-reader -n demo --type=json \
  -p='[{"op":"add","path":"/rules/1/verbs/-","value":"list"}]'

The index depends on the order of rules in that specific Role, so check it with -o yaml first. The check is the same command as at the start: kubectl auth can-i should answer yes. RBAC changes take effect on the very next request, no need to restart the Pod.

If what's missing isn't a verb but the whole RoleBinding:

kubectl create rolebinding app-reader-binding \
  --role=app-reader \
  --serviceaccount=demo:app-sa \
  -n demo

In --serviceaccount, the namespace and SA name are written with a colon. Get the namespace wrong, and you'll hit that same fourth cause. In an existing RoleBinding, you can fix subjects with kubectl edit, but you can't change roleRef: the API will say the field is immutable, and you'll have to create the binding again.

If the Pod is running under the wrong SA, fix the Pod template in the Deployment (spec.template.spec.serviceAccountName). You can't change the SA of a running Pod, so the Deployment will recreate it.

Common mistakes

Granting permissions to SA default. The 403 goes away, but any Pod in the namespace that doesn't have its own SA set gets the permissions too. The right fix is to give the Pod its own SA.

Checking without the prefix or without -n. kubectl auth can-i list configmaps --as=app-sa asks about user app-sa, not an SA, and a no tells you nothing about the real problem.

Granting "just in case". verbs: ["*"] or the built-in edit and cluster-admin roles clear the error, but they also give the SA write access and let it read Secrets. Grant permissions based on the actual Forbidden error, not more. kubectl auth can-i --list before rolling out shows what the SA actually got.

Deciding the fix didn't work. After one fix, a 403 can turn into a different 403: for example, the subject is now correct, but a verb is missing. Read the error from scratch, word by word, every time.

In short

• A Forbidden error contains subject, verb, resource, apiGroup, and namespace: everything you need to fix it.
• Read the subject first: it shows which SA the Pod is actually running as.
• kubectl auth can-i --as=system:serviceaccount:<ns>:<sa> gives a definite answer, --list shows all of a subject's permissions at once.
• Role rules work as a Cartesian product of apiGroups, resources, and verbs; RBAC has no deny rules.
• RBAC changes take effect immediately, no Pod restart needed.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl logs deploy/<name> --tail=1: the fresh error text.
• kubectl auth can-i <verb> <resource> --as=system:serviceaccount:<ns>:<sa> and the --list variant.
• kubectl get role <name> -o yaml, kubectl get rolebinding -o wide.
• kubectl edit or kubectl patch for Role, RoleBinding, and Deployment.