RBAC locked down prod: CI gets Forbidden

The pipeline fails at the deploy step, even though the manifests didn't change. The log has one line:

Error from server (Forbidden): error when creating "deploy.yaml": deployments.apps is forbidden: User "system:serviceaccount:prod:ci-deployer" cannot create resource "deployments" in API group "apps" in the namespace "prod"

The message contains the whole diagnosis: who (ci-deployer), what they wanted (create deployments), in which apiGroup (apps), in which namespace (prod). What's left is to figure out what's missing in the Role, and add exactly that.

This lesson builds on "Forbidden for a ServiceAccount": it covers the four causes of a 403 and kubectl auth can-i. Here it's the same mechanics, but from the CI side, where a single manifest holds resources from several apiGroups.

Concepts

apiGroup is the API group a resource lives in. Core resources (Pod, Service, ConfigMap) have an empty group, written as "". For everything else, the group shows up in the manifest's apiVersion before the slash: apps/v1 means group apps, batch/v1 means batch.

Role rules name a resource with a triple: apiGroups, resources, verbs. A rule with apiGroups: ["apps"] and resources: ["ingresses"] creates without error, but grants nothing: Ingress isn't in group apps. The apiserver doesn't check whether the listed resources exist.

The subject for CI is usually a ServiceAccount: the pipeline gets its token and talks to the API as that account. In errors it's written as system:serviceaccount:<namespace>:<name>.

What CI needs to deploy

kubectl apply -f deploy.yaml isn't one operation. For every object in the file, kubectl does:

1. get, to see whether the object exists;
2. create if it doesn't, or patch if it exists and differs;
3. if the object already matches the manifest, no write request is sent at all (unchanged).

The kubectl rollout status step after apply watches the Deployment, and needs get, list, and watch on deployments.

If the manifest holds a Deployment, Service, and ConfigMap, kubectl makes requests to two different apiGroups: apps and core. Add a CronJob, and a third one shows up, batch. The Role has to cover each group separately.

apply has another property that surprises people in prod: it isn't atomic. Objects from the file apply one by one, and a Forbidden on one doesn't roll back the rest. The pipeline failed, but part of the release is already in the cluster.

configmap/web-config configured
Error from server (Forbidden): error when applying patch:
...
for: "release.yaml": error when patching "release.yaml": deployments.apps "web" is forbidden: User "system:serviceaccount:prod:ci-deployer" cannot patch resource "deployments" in API group "apps" in the namespace "prod"

Here the ConfigMap has already been updated, but the Deployment hasn't. Once permissions are fixed, apply needs to run all the way through again.

The apiGroups map

Resource	apiGroup	Where
Pod, Service, ConfigMap, Secret, Namespace, ServiceAccount	"" (core)	basics
Deployment, ReplicaSet, StatefulSet, DaemonSet	apps	controllers
Job, CronJob	batch	jobs
Ingress, NetworkPolicy	networking.k8s.io	networking
Role, RoleBinding, ClusterRole, ClusterRoleBinding	rbac.authorization.k8s.io	RBAC itself
HorizontalPodAutoscaler	autoscaling	autoscaling

You don't need to memorize the table. The cluster itself shows the group:

kubectl api-resources

NAME                       SHORTNAMES   APIVERSION             NAMESPACED   KIND
...
deployments                deploy       apps/v1                true         Deployment
horizontalpodautoscalers   hpa          autoscaling/v2         true         HorizontalPodAutoscaler
ingresses                  ing          networking.k8s.io/v1   true         Ingress
...

The most common mistake in a fresh Role: apiGroups: [""] for everything, including Deployment. The deploy fails with Forbidden, because Deployment lives in apps, not core. This exact case is what prints deployments.apps is forbidden: the suffix after the dot is the group.

How to reproduce the problem from your own terminal

kubectl auth can-i understands the --as flag and checks permissions as any subject. Exactly what CI would do:

kubectl auth can-i create deployments -n prod \
  --as=system:serviceaccount:prod:ci-deployer

no

The full list of allowed actions:

kubectl auth can-i --list -n prod \
  --as=system:serviceaccount:prod:ci-deployer

If the ServiceAccount has no RoleBinding at all, the list still isn't empty. It keeps the built-in rights every authenticated client has:

Resources                                       Non-Resource URLs   Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                  []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
                                                [/api/*]            []               [get]
                                                [/api]              []               [get]
...

Only these lines and nothing about deployments, services, configmaps: that means there's no RoleBinding, or it points at the wrong ServiceAccount.

Check the apply itself without creating anything:

kubectl apply -n prod -f deploy.yaml --dry-run=server \
  --as=system:serviceaccount:prod:ci-deployer

--dry-run=server sends the request to the server, authorization is checked for real, and the object isn't saved. A convenient way to test permissions before the next pipeline run. Keep in mind point 3 from the list above: if the object in the cluster already matches the manifest, patch isn't sent, and dry-run won't notice that this verb is missing from the Role. Only auth can-i gives an exact answer for each verb.

Minimum permissions to deploy

For a Deployment, Service, and ConfigMap combo, this set is enough:

rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["services", "configmaps"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]

Why exactly this:

• replicasets read-only: the Deployment controller creates ReplicaSets, not CI. You still need to read them, for example for kubectl rollout history;
• pods only needs get/list/watch: CI doesn't create Pods directly, but looks at them when a rollout gets stuck;
• update isn't needed for apply: apply updates an object via patch. It's useful if the pipeline calls kubectl replace, otherwise drop the verb;
• delete is usually out of place in a CI role. Deleting prod resources is a separate, dangerous operation, handled by a different workflow.

Order of recovery

1. Read every Forbidden line from the pipeline log: verb, resource, apiGroup. There can be several lines, one per object.
2. Look at the current Role: kubectl get role -n prod.
3. Read its rules: kubectl get role ci-deployer -n prod -o yaml.
4. Run can-i for every operation from the errors.
5. Add what's missing with a patch, without rewriting the whole Role.
6. Check can-i again.
7. Rerun the deploy end to end: part of it may already have applied.

The temptation at step 5 is strong: grant everything and sort it out later. That path ends in a bigger incident.

Common mistakes

verbs: ["*"]. Grant everything at once. A month later you have a pipeline that can delete anything in the namespace, including prod's own Deployment and its Secret.

resources: ["*"]. The same mistake from the other side. Opens access to Secret, to RBAC objects, to everything. List resources explicitly.

ClusterRole cluster-admin via a RoleBinding. A common excuse: "the RoleBinding is namespaced, so it's limited." Only the namespace is limited. Inside it, the SA can read every Secret, create ServiceAccounts and RoleBindings, and hand that same cluster-admin to any other SA in the namespace. Other namespaces and cluster-level resources (Node, Namespace) stay out of reach.

Right resource, wrong group. horizontalpodautoscalers in a rule with apiGroups: ["apps"], ingresses in core. The Role creates without errors, Forbidden stays. Check the group against kubectl api-resources.

One Role for everything. A big Role with wildcards across every resource can't be audited. Several small Roles with separate RoleBindings for different jobs work better: deployer, reader, secret rotation.

How to avoid hitting this again

Keep CI's permissions in the same repository as the manifests, and roll them out with the same process. A new resource type in the manifests (CronJob, Ingress, HPA) should arrive in the same change as the rule for its apiGroup. You can add a kubectl auth can-i check as a separate pipeline step for the operations you need: then a missing permission shows up before a deploy attempt, not in the middle of a release.

Useful commands

kubectl get role,rolebinding -n prod
kubectl describe role ci-deployer -n prod
kubectl auth can-i --list -n prod \
  --as=system:serviceaccount:prod:ci-deployer
kubectl api-resources

In short

• A Forbidden message names the subject, verb, resource, apiGroup, and namespace. Read every line in full, the whole diagnosis is there.
• The apiGroup shows up in the manifest's apiVersion and in the error's suffix (deployments.apps, cronjobs.batch). Core is written as "".
• kubectl apply does get, then create or patch, and applies objects one by one: on Forbidden, part of the release is already in the cluster.
• kubectl auth can-i --as= reproduces CI's permissions from your own terminal, apply --dry-run=server --as= checks requests without saving anything.
• verbs: ["*"] and resources: ["*"] have no place in prod: a minimal set of verbs and an explicit list of resources.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl auth can-i <verb> <resource> --as=system:serviceaccount:<ns>:<sa> and the --list variant.
• kubectl get role <name> -o yaml and kubectl api-resources.
• kubectl patch role <name> --type=json or kubectl edit role <name>.
• For the mini-project, everything from the module: kubectl get pods, kubectl describe pod (Events section), kubectl logs, kubectl get endpointslice, kubectl set image, kubectl patch.