StorageClass and dynamic disk provisioning

provisioner, reclaimPolicy, and volumeBindingMode: three fields that decide how and when a PVC gets a real disk, and what happens to it afterward.

Why you need a StorageClass

A PersistentVolumeClaim (PVC) is a request for storage: "give me 1 GiB in ReadWriteOnce mode (the volume is mounted for writing by only one node)". StorageClass decides how that storage shows up.

Without a StorageClass, an administrator would have to create a PersistentVolume (PV) by hand for every request. A PV is the actual piece of storage. With a class, the PV is created automatically when a PVC comes in. That's what dynamic provisioning means.

StorageClass: the disk recipe

StorageClass is a cluster object that describes how disks get made. It lives at the whole-cluster level, not in a namespace. It has three key fields.

Field	What it decides	Value of standard in kind
provisioner	which plugin creates the PV	rancher.io/local-path
reclaimPolicy	the disk's fate after the PVC is deleted	Delete
volumeBindingMode	when the PV gets created	WaitForFirstConsumer

This is what the local-path class from k3s and k3d looks like. In kind, the same class is called standard:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

To see your cluster's classes:

kubectl get storageclass

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  2m

The resource's short name is sc: kubectl get sc prints the same thing.

provisioner

provisioner is the plugin that creates a PV for a PVC request. In small local clusters, the provisioner is usually called rancher.io/local-path and creates a directory on the node. In AWS it's ebs.csi.aws.com. The right plugin has to be running in the cluster, or the PVC will hang in Pending.

Old names like kubernetes.io/aws-ebs or kubernetes.io/gce-pd still show up in ancient manifests. These built-in cloud-disk provisioners were cut out of Kubernetes' code. Today, cloud and network storage connects through a CSI driver, and its name goes in the provisioner field (docs). CSI (Container Storage Interface) is the standard interface between Kubernetes and a storage driver.

reclaimPolicy

reclaimPolicy sets what happens to the PV after the PVC is deleted.

• Delete - the PV and the disk under it are deleted. This is the default: if the field isn't set in the class, it's Delete.
• Retain - the PV and the disk stay. Safer for important production data.

The class passes its value to every PV it creates. On the PV, it shows up in the RECLAIM POLICY column of kubectl get pv.

Retain: what happens to the PV next

After the PVC is deleted, a disk with Retain moves to Released status:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM            STORAGECLASS   ...
pvc-8d0b6c1e-2f7a-4f35-9a0e-5b1c3d2e4f60   64Mi       RWO            Retain           Released   default/keepme   keep           ...

Released means "the data is intact, but the disk remembers its previous owner". The reference to the deleted PVC is stored in the PV's spec.claimRef field, along with its uid. A new PVC, even with the same name, gets a new uid. That's why Kubernetes never hands such a disk to anyone on its own: someone else's data shouldn't accidentally end up with another app.

What to do with a Released disk is a human decision (docs). You can copy the data off and delete the PV. Or you can hand the disk to a new PVC: clear the PV's reference to the previous owner, after which the PV becomes Available. A PVC asks for a specific PV through the spec.volumeName field (docs). Without volumeName, the PVC gets a new empty disk from the provisioner.

volumeBindingMode

volumeBindingMode sets when the PV gets created.

• Immediate - the PV is created right after the PVC appears. This is the default if the field isn't set. On a cluster with several zones, the PV might end up in a zone different from where the Pod will land.
• WaitForFirstConsumer - the PV is created once a Pod that uses the PVC shows up. That's how local-path and standard are set up in kind.

Because of WaitForFirstConsumer, a PVC without a Pod stays Pending. It will only move to Bound once the scheduler picks a node for the Pod. A Pod is the smallest unit that Kubernetes runs.

Default StorageClass

One StorageClass in the cluster is usually marked as the default class with the annotation storageclass.kubernetes.io/is-default-class: "true". In kubectl get sc, it shows (default) after its name. If the PVC doesn't specify a storageClassName field, this class is used.

If there's no default class and the field is empty, the PVC stays Pending, and its events show:

Normal  FailedBinding  ...  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

A Pod with such a PVC also hangs in Pending. You can only see the cause in kubectl describe pvc, so when a Pod is Pending, check its events first, then the PVC's events.

When a default class shows up again, Kubernetes writes it into PVCs that don't have storageClassName set, and they bind without being recreated. This is retroactive default assignment, enabled since Kubernetes 1.28 (docs).

This doesn't happen instantly. In kind, the class showed up in the PVC after about 40 seconds. But the Pod that was already Pending only got another try from the scheduler about five minutes later. You don't have to wait: delete that Pod, and the Deployment will create a new one, which gets a disk right away.

If there are several default classes, a new PVC without a class gets the most recently created one. It'll work, but nobody likes guessing which disk they'll end up with. Keep exactly one default class in the cluster.

An explicit storageClassName

A PVC can name its class directly:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-fast
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 1Gi

An explicit storageClassName removes the dependency on whichever class the cluster currently has marked as default. Write the name exactly as it is, character for character. An invalid name like local_path (underscores aren't allowed in object names) gets rejected by the server right away. But a typo like standart instead of standard slips past kubectl apply: the PVC gets created and hangs in Pending, with this in its events:

Warning  ProvisioningFailed  ...  persistentvolume-controller  storageclass.storage.k8s.io "standart" not found

An empty string storageClassName: "" means "no class". Such a PVC waits for a PV without a class, created by hand by an administrator, and the default class doesn't apply to it.

A class isn't changed after it's created

An existing StorageClass can't have its provisioner, parameters, reclaimPolicy, or volumeBindingMode changed. The server responds with an error:

The StorageClass "now" is invalid: reclaimPolicy: Forbidden: updates to reclaimPolicy are forbidden.

To change the behavior, the class is deleted and created again. PVs already created don't change: each has its own copy of reclaimPolicy. Annotations and labels on the class can be changed, that's how you switch the default class.

Common mistakes

• Copying a class name from someone else's manifest (gp2, local-path) into a cluster that doesn't have that class. The PVC silently hangs in Pending.
• Expecting Bound without a Pod on a class with WaitForFirstConsumer.
• Leaving the cluster without a default class, or with two of them.
• Keeping a database on a class with Delete and deleting the PVC "for a moment". The data goes away with it.
• Expecting a new PVC with the same name to pick up a Released disk on its own.
• Editing reclaimPolicy on an existing class instead of creating a new one.

Useful commands

kubectl get sc
kubectl describe sc standard
kubectl get pv
kubectl describe pvc <pvc-name>
kubectl explain sc
kubectl explain pvc.spec.storageClassName

What you'll need for the tasks

• kubectl get storageclass and the (default) mark.
• kubectl describe pvc <name> and the Events block.
• The storageclass.kubernetes.io/is-default-class annotation.
• A StorageClass manifest with provisioner, reclaimPolicy, volumeBindingMode.
• A PVC with storageClassName and volumeName, a volume in a Pod (the PVC lesson).
• kubectl get pv: the Bound, Released, and Available statuses, and the spec.claimRef field.

The tasks live in the labs/ folder: each lab brings up its own kind cluster on your machine, instructions in its README.md. The last lab, a mini-project, pulls together the whole "Storage" module.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-storage-class     # set up the starting state
./run.sh verify k8s-storage-class    # check the current step

In short

• StorageClass is a cluster-level object, a recipe for making disks. It doesn't live in a namespace.
• provisioner names the plugin that makes the PV. No plugin in the cluster, PVC hangs in Pending.
• reclaimPolicy: Delete removes the disk along with the PVC, Retain leaves it in Released status until a human decides.
• volumeBindingMode: WaitForFirstConsumer delays creating the PV until a consuming Pod shows up.
• A PVC without storageClassName takes the default class. No default class, PVC won't bind until one appears.
• A class's fields can't be changed after creation; you can only recreate the class.