CRD: your own resource type in the cluster

Why you need your own resource type

The number of built-in types in Kubernetes is finite: Pod, Service, ConfigMap, and a few dozen more. When an application needs an object that isn't on that list, it gets added through a CustomResourceDefinition.

A CustomResourceDefinition (CRD) is a definition of a new resource type. Once a CRD is applied, the cluster starts understanding a new kind, and the same kubectl get, kubectl apply, kubectl describe, the same RBAC model, and the same manifest format work with it.

That's how almost all third-party Kubernetes software works. cert-manager adds the Certificate type, Prometheus Operator adds ServiceMonitor, Argo CD adds Application. None of them patches the Kubernetes source code. They all register their own CRDs, and the cluster's API extends on the fly.

You can check which types the cluster knows about right now with one command:

kubectl api-resources
kubectl api-resources --api-group=cert-manager.io

The second command shows only the types in the cert-manager.io group. If cert-manager isn't installed in the cluster, it prints an empty table with one header row, which isn't an error.

Everything below repeats on any cluster where you have admin rights, including a local kind cluster from the "Kubernetes locally" track. In this lesson's labs, a setup script brings up the cluster.

CRD and CR are different things

The two terms are easy to mix up, and the difference matters.

• CRD is the type definition. It describes once that, say, a Widget now exists in the cluster.
• CR (CustomResource) is a specific object of that type. There can be many of them: widget-a, widget-b.

A programming analogy: CRD is like a class, CR is an object of that class. You register the type first, then create objects. The permissions differ too: creating a CRD needs access to the whole cluster, creating a Widget only needs permissions in one namespace.

CRD anatomy

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.demo.example.com
spec:
  group: demo.example.com
  scope: Namespaced
  names:
    plural: widgets
    singular: widget
    kind: Widget
    shortNames: [wd]
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                color:
                  type: string

group

group is the API namespace a type belongs to. For built-in resources it's either an empty string (Pod, Service) or a short name (apps for Deployment). For your own types, you take a domain name so you don't collide with someone else's: demo.example.com.

name and names

metadata.name must be in the form <plural>.<group>, so widgets.demo.example.com. If you write it any other way, apiserver returns an error and the CRD doesn't get created:

The CustomResourceDefinition "widget.demo.example.com" is invalid: metadata.name: Invalid value: "widget.demo.example.com": must be spec.names.plural+"."+spec.group

The names block sets how the type is addressed in commands:

• kind: Widget — the name in the manifest (the kind field).
• plural: widgets — the form for kubectl get widgets.
• singular: widget — the form for kubectl get widget my-widget.
• shortNames: [wd] — a short alias, kubectl get wd.

scope

scope determines where objects of this type live.

• Namespaced — the object belongs to a namespace, like a Pod. kubectl get widgets -n team-a shows only that namespace's widgets.
• Cluster — the object is shared across the whole cluster, like a Node or the CRD itself.

The CRD itself is always cluster-scoped: it changes the API of the whole cluster. So only someone with cluster-wide permissions can create one, and that's why installing operators in managed clusters is usually an admin's job, not the development team's.

versions and schema

versions is the list of the type's API versions. Exactly one of them is marked storage: true: that's the format the object is stored in in etcd. served: true means the version is served through the API. When there are several versions, apiserver converts the object between them, and for non-trivial cases you wire up a conversion webhook.

schema.openAPIV3Schema describes the object's fields. Without a schema, apiserver won't accept the CRD: in apiextensions.k8s.io/v1 it's required.

The CustomResourceDefinition "widgets.demo.example.com" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value

The schema acts as validation: if you declare size as integer, a string won't get written there.

The Widget "my-widget" is invalid: spec.size: Invalid value: "string": spec.size in body must be of type integer: "string"

An object of the new type

After a CRD is applied, the cluster needs a few seconds to register the type. Readiness shows up as the Established condition:

kubectl get crd widgets.demo.example.com
kubectl wait --for=condition=Established crd/widgets.demo.example.com --timeout=30s

From there, the object is created like a regular manifest. apiVersion is assembled from group and version, kind comes from names.kind:

apiVersion: demo.example.com/v1
kind: Widget
metadata:
  name: my-widget
spec:
  size: 3
  color: blue

kubectl apply -f widget.yaml
kubectl get widgets
kubectl get widget my-widget -o jsonpath='{.spec.size}'

What else the schema does

The schema isn't only for validation; the rest of the work with the type grows out of it too.

kubectl explain reads the same schema and explains the custom type's fields the same way it explains a Pod's fields:

kubectl explain widget.spec
kubectl explain widget.spec.size

Fields that aren't in the schema get stripped by apiserver on write. That's called pruning, and it's on by default in apiextensions.k8s.io/v1.

How that looks depends on the client. kubectl 1.27 and newer asks apiserver for strict field validation by default (--validate=strict), and an unknown field turns into an error:

strict decoding error: unknown field "spec.owner"

With --validate=warn you'd only get Warning: unknown field "spec.owner", with --validate=false nothing at all. Older kubectl and many client libraries stay just as quiet. In those cases apply goes through with no error, and after kubectl get -o yaml the field is missing from the object. If you actually want arbitrary keys, you allow them with x-kubernetes-preserve-unknown-fields: true.

The columns in kubectl get output are configurable too. By default, only the name and age are shown. additionalPrinterColumns inside a version sets your own set, and then you also have to describe the AGE column yourself if you want it:

versions:
  - name: v1
    served: true
    storage: true
    additionalPrinterColumns:
      - name: Size
        type: integer
        jsonPath: .spec.size
      - name: Color
        type: string
        jsonPath: .spec.color

spec, status, and subresources

Built-in resources have a split: spec is what the person wants, status is what's actually happening. For custom types, you turn that split on explicitly:

subresources:
  status: {}

After that, status lives as a separate subresource. Writing to the main object doesn't change status, and writing to status doesn't touch spec. The practical effect: a person edits spec through kubectl apply, a controller writes status through a separate call, and they don't stomp on each other's work. Trying to write status with a plain kubectl patch ends in patched (no change): the subresource needs the --subresource=status flag.

The second subresource, scale, lets you run kubectl scale and HorizontalPodAutoscaler against a custom type, as long as the schema has fields for a replica count.

The controller: what turns a record into action

A CRD by itself only stores data. The Widget object sits in etcd, and without external code nothing happens when it appears.

A controller adds the action: a process that's subscribed to events on objects of its type and drives the cluster toward the described state. It runs a reconcile loop:

1. Read the desired state from the object's spec.
2. Look at the actual state in the cluster.
3. Take steps that close the gap.
4. Write the result to status.

An important property of this loop: it's level-triggered, meaning it looks at the current state, not at the list of events that happened. One missed event breaks nothing; the next pass will still see the mismatch. That puts a requirement on reconcile: it has to be idempotent. Running it again on an unchanged object shouldn't create anything twice.

A controller like this usually also attaches a finalizer to the object — an entry in metadata.finalizers that stops the object from being deleted for good until the controller has cleaned up its external resources. While the finalizer is in place, the object stays in Terminating after kubectl delete.

The operator

The "CRD + controller" combo is called an operator. An example is cert-manager: it registers Certificate, the controller sees the new object, goes to Let's Encrypt, puts the certificate in a Secret, and writes the expiry date to status. Operators are written on top of frameworks like controller-runtime, Kubebuilder, and Operator SDK, which take care of the boilerplate and leave the developer with the reconcile function.

A CRD without a controller is still useful: it's a way to store configuration in a kubectl-compatible form, with schema validation and ordinary RBAC. But until a controller shows up, it's a database, not automation. Why operators came to exist, which ones show up in production, and what they cost is covered in the "Operators" lesson of this track.

What usually breaks

• metadata.name doesn't match <plural>.<group>. apiserver rejects the CRD with an error about the name mismatch.
• A field isn't described in the schema. A recent kubectl answers unknown field, a client without strict validation silently loses the field on write. In both cases, you fix the CRD's schema, not the manifest.
• The CRD got deleted to recreate it with a fixed schema. Every object of that type went with it. Change the schema in place instead: kubectl apply or kubectl edit on the CRD.
• The object is applied in the same file as the CRD, and you get no matches for kind.
• Not enough permissions: a CRD is cluster-scoped. You can create a Role in a namespace with permission on customresourcedefinitions, but it won't actually work; you need a ClusterRole.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-crd     # set up the starting state
./run.sh verify k8s-crd    # check the current step

In short

• A CRD registers a new kind, after which ordinary kubectl commands and ordinary RBAC work with it.
• CRD is the type definition, CR is an object of that type; a CRD is always cluster-scoped.
• metadata.name is strictly <plural>.<group>, a schema is required in the v1 version, fields outside the schema get stripped.
• subresources.status separates the person's intent from the controller's result.
• A controller runs a reconcile loop: it compares spec with reality and closes the gap; a CRD paired with such a controller is an operator.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl get crd, kubectl get crd <name> -o yaml, kubectl explain <kind>.spec.
• CRD anatomy: metadata.name as <plural>.<group>, scope, names, versions[].schema.openAPIV3Schema, additionalPrinterColumns, subresources.
• kubectl apply -f, kubectl wait --for=condition=Established crd/<name>.
• Reading apiserver errors: unknown field, must be of type integer.