Operators: why they exist and what they cost

Where Deployment stops

Deployment covers two jobs. A Pod dies, it gets recreated. You change the image tag, it rolls out the new version in batches, with the old Pods standing by until the new ones become Ready.

That's where the boundary starts. Deployment doesn't know what's running inside the container. To it, Postgres and a static nginx look the same: a process that either answers its probes or doesn't.

Because of that, it can't do any of what actually running a database in production involves:

• promote a replica to master when the master stops responding;
• take a backup and verify it restores;
• roll out a major version that needs an on-disk schema migration;
• recreate users and roles after a restore;
• switch application traffic to the new master after a failover.

All of this is automatable too, the automation just doesn't live in the cluster. It lives in the on-call engineer's head, in a runbook on the wiki, or in a set of scripts a person runs. At three in the morning, a person works slowly and makes mistakes.

What an operator does

An operator moves this knowledge into code that runs in the cluster all the time. It's subscribed to events on its objects and to cluster state, and it reacts without a human.

Without the magic: an operator is just a controller that does the same thing an on-call engineer would do by the book. The same API calls, the same checks, the same order of steps. The difference is it doesn't sleep, doesn't mix up environments, and executes the steps the same way every time.

The mechanics here are already familiar. An operator registers its resource type through a CRD, and inside it runs a reconcile loop: reads spec, looks at the actual state, closes the gap, writes status. This loop, level-triggered behavior and finalizers are covered in the "CRD" lesson of this track.

One example: a Postgres master failing

What this looks like by hand. Monitoring sends an alert that the master isn't responding. The on-call engineer logs into the replicas, checks which one is furthest ahead in the WAL, picks it as the new master, and runs promote. Then they repoint the remaining replicas at the new master, fix the endpoint the application talks to, and check that writes go through. Between the alert and writes working again, anywhere from a few minutes to half an hour passes.

The operator runs the same sequence. It's already polling the instances constantly, so it sees the failure itself. Then it compares replication positions, picks a candidate, promotes it, repoints the other replicas at the new master, and switches the Service clients go through to the new Pod. The order of steps is baked into the code and doesn't depend on who's on call today.

The human only describes intent. A custom resource looks roughly like this:

apiVersion: db.example.com/v1
kind: PostgresCluster
metadata:
  name: orders
spec:
  instances: 3
  version: "16"
  storage:
    size: 100Gi
  backup:
    schedule: "0 2 * * *"
    retentionDays: 14

This is a generalized shape, not a manifest for a real product. Every operator has its own set of fields: some describe backups as a separate resource, some set the version through the whole image, some have a block for connecting to external S3. Before applying anything, read that specific operator's documentation and its schema through kubectl explain.

The landscape: what you actually run into

Operators stopped being exotic a while ago. Chances are your cluster already has two or three installed.

• cert-manager. Adds Certificate and Issuer. Takes over issuing and renewing TLS certificates: it runs the ACME challenge itself, puts the certificate into a Secret, and renews it before it expires.
• Prometheus Operator. Adds ServiceMonitor, PodMonitor, PrometheusRule, Alertmanager. Removes manual edits to Prometheus config: the list of scrape targets and alert rules get described as ordinary manifests next to the application.
• CloudNativePG. Adds Cluster and backup resources. Runs an entire Postgres cluster: replication, failover, backups to object storage, point-in-time recovery.
• Zalando postgres-operator. A different operator for the same job, with its own type postgresql and its own approach to configuration.
• Strimzi. Adds Kafka, KafkaTopic, KafkaUser. Takes over deploying brokers, rebalancing, and managing topics and access rights.
• Argo CD. Adds Application and AppProject. Keeps the cluster's state matching a git repository, which removes manual kubectl apply during rollouts.

Versions and specific field sets are deliberately left out here: they change from release to release, and the only reliable source is the project's documentation at the time you install it.

Maturity levels

The Operator Framework has a scale called Operator Capability Levels, five levels:

1. Basic install. The operator can deploy the application from a custom resource and configure it.
2. Upgrades. Can move the application to a new version, including any migrations it requires.
3. Full lifecycle. Backups, restore, fault-tolerant behavior when an instance is lost.
4. Metrics and alerts. Exposes the application's state, metrics, and ready-made alert rules.
5. Auto pilot. Autoscaling, self-tuning parameters, anomaly detection and response.

The scale exists for one practical question. "There's an operator for our database" does not mean that something will happen automatically when the master fails. A level-2 operator will deploy the cluster and handle upgrades, but failover and backups stay on you. Before you throw out the runbook, check exactly what the operator can do, and check it on a test cluster rather than by reading the README.

When you don't need an operator

An operator earns its place where there's state and nontrivial operations on it: databases, message brokers, search engines, clusters with leader election.

If the application is stateless, and the whole job comes down to rolling out one set of manifests with different values per environment, an operator buys you nothing. A Kustomize overlay or a Helm chart is enough (track "Packaging applications"). Templating plus kubectl apply covers these cases completely, and later you'll be debugging your own YAML, not someone else's code.

A simple signal: if the application has no operations beyond "roll out a new version", you don't need to write an operator for it.

is there state and operations on it?
  yes  -> operator
  no   -> Helm or Kustomize

The price of an operator

An operator isn't free, and the bill comes in three parts.

Permissions. It's another Pod in the cluster, usually with broad permissions: at minimum, full access to its own types in every namespace, often plus access to Secret, PersistentVolumeClaim and Pod in the target namespaces. A database operator can, by definition, delete and recreate your database. Compromising that Pod means compromising the data. Permissions get granted through a ServiceAccount, ClusterRole and ClusterRoleBinding, and before installing an operator you read what rules it's asking for. A binding to the built-in role cluster-admin means permissions on everything in the cluster.

Dependency. An operator is someone else's project, with its own release pace. If development stalls and you're running production on it, you've got an abandoned component in prod that will sooner or later stop building against a new Kubernetes version. Before picking one, look at how active the repository is and who's behind the project.

Debugging. When a database won't come up, you have to dig on two levels at once: in Postgres itself, and in someone else's reconcile logic. Why the controller decided the state was reached. Why it's stuck in a recreate loop. It's an extra layer between you and the application, and it adds time to every incident.

How they're installed and inspected

Operators get installed either as a Helm chart or as one combined manifest from the project's repository. Installing one usually needs cluster-wide permissions: its CRDs get applied along with it, and CRDs are cluster-scoped.

After installing, check that the types appeared:

kubectl get crd
kubectl api-resources --api-group=cert-manager.io

Next, look at the operator itself and its objects:

kubectl get pods -n cert-manager
kubectl logs -n cert-manager deploy/cert-manager -f
kubectl describe certificate my-cert
kubectl get certificate my-cert -o yaml

The main source of truth for a custom object is its status and the events in kubectl describe. That's where the controller writes what worked and what didn't. The operator Pod's logs matter when status is empty or silent.

If the operator lacks permissions, the apiserver answers with a 403, and a line like this shows up in status or the logs:

secrets is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot create resource "secrets" in API group "" in the namespace "default"

You can check exactly what the operator's ServiceAccount is allowed to do without reading through every role:

kubectl auth can-i --list --as=system:serviceaccount:cert-manager:cert-manager
kubectl auth can-i get secrets -n default --as=system:serviceaccount:cert-manager:cert-manager

There are also catalogs: OperatorHub collects public operators along with their maturity level, and Operator Lifecycle Manager (OLM) installs and upgrades them by subscription while tracking version compatibility. In clusters without OLM, operators just get installed by hand, and that's a normal setup.

Who owns the objects

A Deployment, ConfigMap and other objects an operator created are usually tagged with a reference back to its custom object, in metadata.ownerReferences. You can look at it like this:

kubectl get deploy orders -o jsonpath='{.metadata.ownerReferences[0].kind}'

That reference has two consequences. First: delete the custom object, and the Kubernetes garbage collector deletes everything it owns along with it. Second: the child object now belongs to the operator, and editing it by hand is pointless. Reconcile is level-triggered, so on the next pass the operator sees the mismatch against its object's spec and reverts it. A manual kubectl scale or kubectl edit on such a Deployment lasts a few seconds. What you change is the custom object's spec.

What usually breaks

• The object got created, and nothing happens. Start with kubectl describe on the object and its status, then kubectl logs for the operator. A common cause is a 403 from the apiserver: the operator's ServiceAccount is missing a verb in its ClusterRole.
• Someone "fixed" the operator's permissions with a binding to cluster-admin. It'll work, but anyone who gets that Pod gets the whole cluster.
• Someone edits the child Deployment by hand, and the edit reverts within seconds. Change the value in the custom object's spec instead.
• Someone deleted the custom object to "restart" the site or database, and the garbage collector took everything it owned along with it.
• Someone deleted a CRD to update its schema. Every object of that type disappears along with the CRD, and their child objects follow.
• Someone trusted that the operator handles failover without checking its maturity level or testing a failure on a test cluster.

In short

• Deployment restarts Pods and rolls out new versions, but doesn't know what's inside the application: failover, backups and schema migrations are outside its job.
• An operator is a controller that performs the on-call engineer's runbook steps, only around the clock and consistently.
• A human describes intent in a custom resource; the fields are different for every operator, and you look them up in the documentation and kubectl explain.
• Operator Capability Levels show how far an operator takes the automation; having an operator doesn't guarantee failover.
• A stateless application is fine with Helm or Kustomize; an operator is for when there's state and operations on it.
• You pay for an operator with broad permissions in the cluster, dependency on someone else's project, and harder incident debugging.

What you'll need for the tasks

In the labs, a training operator website-operator runs with the type Website: for a Website object, it maintains a ConfigMap with the page and a Deployment with the web server in the same namespace, and writes the result to status. You'll use:

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl get websites -A, kubectl describe website <name>, kubectl get website <name> -o yaml: the object's spec and status.
• kubectl -n website-system logs deploy/website-operator --tail=20.
• kubectl get clusterrole <name> -o yaml, kubectl get clusterrolebinding -o wide, kubectl auth can-i ... --as=system:serviceaccount:<ns>:<sa>.
• kubectl edit, kubectl patch --type=merge, kubectl apply -f.
• From earlier lessons in the track: editing a CRD's schema in place, and moving an object to another namespace.