PersistentVolumeClaim: the volume that outlives the Pod

How a PVC and a PV give an app a persistent disk, what accessModes means, and why a PVC stays Pending without a Pod.

Why you need persistent storage

A Pod is the smallest unit Kubernetes runs. A container runs inside the Pod. By default a Pod doesn't keep data for long. The container crashes, kubelet starts a new one, and the files on the old one's filesystem are gone. kubelet is the Kubernetes agent on a node that runs containers.

An emptyDir volume survives a container crash, but lives only as long as the Pod: delete the Pod, and the data goes with it (documentation).

That doesn't work for a database or a queue. They need storage that outlives the Pod being deleted. Two objects handle that: PVC and PV.

PVC and PV: request and resource

PersistentVolumeClaim (PVC) is a request for storage from an app. It states how much space is needed and in which access mode.

PersistentVolume (PV) is a provisioned chunk of storage in the cluster.

A PVC and a PV bind, and after that the Pod mounts the PVC like an ordinary directory. The PVC asks, the PV provides.

A PV is usually created automatically. A provisioner handles that: a storage plugin tied to a StorageClass. In small local clusters that's local-path (k3s) or standard (kind, minikube); in AWS it's the EBS CSI driver (ebs.csi.aws.com). Kubernetes itself doesn't create cloud disks. The cloud-disk plugins that used to be built into Kubernetes's code have been removed, replaced by CSI drivers (documentation). CSI (Container Storage Interface) is the standard interface Kubernetes uses to talk to a storage driver.

In kind the default class is called standard, and its provisioner rancher.io/local-path creates a PV as a directory on the node's disk:

kubectl get storageclass

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  2m

Minimal YAML

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

No StorageClass is named here, so the default class is used. The field resources.requests.storage sets the size. A PVC lives in a namespace, same as a Pod.

accessModes: the access mode

accessModes sets how many nodes can open the volume at once, and how.

• ReadWriteOnce (RWO): one node mounts it for writing. Several Pods on that node can share the volume. The usual choice for block storage like AWS EBS through a CSI driver.
• ReadOnlyMany (ROX): several nodes, read-only.
• ReadWriteMany (RWX): several nodes, for writing. Only works on network filesystems (NFS, CephFS).
• ReadWriteOncePod (RWOP): exactly one Pod in the whole cluster mounts it for writing. Only supported by CSI volumes.

Details and a support table by volume type are in the documentation.

The API only checks that the mode is on this list. A typo like ReadWriteOne is rejected right away:

The PersistentVolumeClaim "typo-pvc" is invalid: spec.accessModes: Unsupported value: "ReadWriteOne": supported values: "ReadOnlyMany", "ReadWriteMany", "ReadWriteOnce", "ReadWriteOncePod"

Whether the storage actually supports a valid mode is decided by the provisioner after the PVC is created. kubectl apply goes through without errors, and the answer shows up in the PVC's events.

Once a PVC is created, its accessModes and storageClassName can't be changed: get it wrong, and you delete it and create it again. The size can be expanded, though, if the StorageClass has allowVolumeExpansion: true. Then it's enough to raise resources.requests.storage; you can't shrink it. The standard class in kind has expansion turned off (ALLOWVOLUMEEXPANSION false in the output above).

WaitForFirstConsumer: the PVC waits for a Pod

A StorageClass sets when the PV is created through the volumeBindingMode field. Local classes usually have WaitForFirstConsumer. The PV isn't created right away, only once a Pod that uses the PVC shows up.

So a PVC under WaitForFirstConsumer stays in status Pending until a Pod references it. Once the Pod starts, the PVC moves to Bound. Waiting for Bound without a Pod is pointless.

kubectl get pvc data-pvc

NAME       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-pvc   Pending                                      standard       <unset>                 10s

kubectl describe pvc data-pvc at this point shows a WaitForFirstConsumer event with the text waiting for first consumer to be created before binding.

Attaching to a Pod

spec:
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-pvc
  containers:
    - name: app
      volumeMounts:
        - name: data
          mountPath: /data

volumes declares the source through claimName (the PVC's name). volumeMounts attaches it to the container at path mountPath. The name in volumeMounts.name must match the name in volumes. If it doesn't, the server rejects the Pod:

The Pod "badmount" is invalid: spec.containers[0].volumeMounts[0].name: Not found: "storage"

A PVC outlives a Pod deletion

A PVC lives separately from the Pod. Deleting the Pod doesn't touch the PVC or the data on disk. You can recreate the Pod and mount the same volume again. To delete the data, the PVC has to be deleted explicitly with kubectl delete pvc data-pvc.

While a Pod is using the PVC, deletion doesn't complete: the PVC hangs in status Terminating, held by the finalizer kubernetes.io/pvc-protection. Delete the Pod, and the PVC is deleted right after (documentation).

Check the state of storage like this:

kubectl get pvc
kubectl get pv
kubectl describe pvc data-pvc
kubectl get storageclass
kubectl get sc -o jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}'

The last command prints the name of the default class.

Common mistakes

• Waiting for Bound before a Pod exists under WaitForFirstConsumer, and looking for a failure that isn't there.
• Looking only at the Pod. If the Pod is Pending because of a volume, the cause is usually in the PVC's events: kubectl describe pvc <name>, the Events block.
• Trying to change accessModes or storageClassName on an existing PVC. The server replies that spec is immutable except for size. It has to be recreated.
• Different name in volumes and volumeMounts.
• Deleting a PVC "to free up space" and forgetting that the data goes with it. For classes with reclaimPolicy: Delete, the disk is deleted along with the PVC.

Useful commands

kubectl explain pvc.spec
kubectl explain pvc.spec.accessModes
kubectl get sc
kubectl get pv
kubectl describe sc <class-name>

What you'll need for the tasks

• A PVC manifest: accessModes, resources.requests.storage.
• volumes blocks with persistentVolumeClaim.claimName, and volumeMounts with mountPath.
• kubectl get pvc, kubectl describe pvc, and the Events block.
• kubectl get storageclass, to find the default class and its VOLUMEBINDINGMODE.
• kubectl exec <pod> -- cat <file> to check data on the volume.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your machine, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-pvc     # set up the starting state
./run.sh verify k8s-pvc    # check the current step

In short

• A PVC is a request for storage, a PV is the provisioned chunk itself. Their link is called a bind.
• A provisioner creates the PV from a StorageClass. Kubernetes creates cloud disks through CSI drivers.
• accessModes and storageClassName can't change after a PVC is created; the size expands only with allowVolumeExpansion: true.
• Under volumeBindingMode: WaitForFirstConsumer, a PVC hangs in Pending until a Pod references it.
• In a Pod, a volume is attached through volumes.persistentVolumeClaim.claimName and volumeMounts.mountPath with a matching name.