StatefulSet and headless Service

A controller for databases and other apps where every replica has its own name and its own disk: the four guarantees of StatefulSet, per-Pod DNS names through clusterIP: None, and a PVC template for each replica.

Why StatefulSet exists

Deployment is a controller that keeps a fixed number of identical Pods running. Pod is the smallest unit Kubernetes runs. Under a Deployment, Pods are interchangeable. One dies, the controller starts a new one with a random name, and it makes no difference. That fits stateless services (no state or disk of their own): nginx, a REST API, a frontend.

A database doesn't work that way. A Postgres replica has its own files on disk, and etcd cluster replicas know each other by name. If a replica comes back with a different name and an empty disk, the rest of the cluster sees a stranger with no data.

StatefulSet is a controller for apps where every replica needs a permanent attachment: a stable name, its own disk, and its own network name. It gives four guarantees.

• Pod names are fixed and numbered: db-0, db-1. The number is called the ordinal index.
• They start in order: db-1 is created only after db-0 becomes Ready. This is the default behavior, podManagementPolicy: OrderedReady. The value Parallel starts all Pods at once (docs).
• Each Pod has its own permanent PVC. A PVC (PersistentVolumeClaim) is a request for storage. db-0 always mounts its own disk, and so does db-1.
• A stable DNS name for each Pod.

Databases (Postgres, MySQL), queues (Kafka), and coordinators (etcd) all need this.

Headless Service

Service is an object that gives a group of Pods one network address. A regular Service hands out one cluster IP and load-balances requests across the Pods. Which Pod answers doesn't matter.

StatefulSet needs to talk to a specific Pod. For that, you make the Service headless (no single cluster IP): set clusterIP: None. Then DNS for the Service's name returns the addresses of every Pod individually, and StatefulSet adds its own name to each Pod:

db-0.db.<namespace>.svc.cluster.local
db-1.db.<namespace>.svc.cluster.local

An app inside the same namespace can use the short form db-0.db: the search list in the Pod's /etc/resolv.conf fills in the rest. The nslookup utility from the busybox image doesn't use that list, so give it the full name. Manifest for a headless Service:

apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  clusterIP: None
  selector:
    app: db
  ports:
    - port: 5432
      protocol: TCP

In kubectl get service, such a Service shows None in the CLUSTER-IP column.

Per-Pod names only work if the StatefulSet's serviceName field points to this Service. The link is by string: the Service's name and the serviceName value must match exactly. If they drift apart, names like db-0.db never show up. There's no error on either StatefulSet or Service.

The Kubernetes docs require a headless Service for StatefulSet specifically (StatefulSet, Limitations section). On clusters running CoreDNS, including kind, the full name db-0.db.default.svc.cluster.local resolves even with a regular Service, but don't rely on that. The real difference shows up on the Service's own name. A regular one hands out a single virtual IP. A headless one hands out the addresses of every ready Pod, and the database's replicas find each other from that list.

volumeClaimTemplates

volumeClaimTemplates is a PVC template for StatefulSet. It's not one shared disk for everyone. From this template, each replica gets its own PVC.

volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ReadWriteOnce]
      resources:
        requests:
          storage: 1Gi

With replicas: 2, the controller creates two PVCs. A PVC's name is built from the formula <template-name>-<sts-name>-<ordinal>. For the template data and StatefulSet db, that's data-db-0 and data-db-1.

db-0 ─┬─ PVC  data-db-0
      └─ DNS  db-0.db
db-1 ─┬─ PVC  data-db-1
      └─ DNS  db-1.db

The volume mounts into the container by the template's name. volumeMounts.name: data attaches PVC data-db-0 to Pod db-0.

A PVC gets its disk from the cluster's StorageClass. In kind, that's standard: it creates a directory on the node and hands it out as a volume. In the cloud, the same manifest gets a network disk from the provider.

Full minimal manifest

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 2
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: db
          image: busybox:1.36.1
          command: ["sh", "-c", "echo running; sleep 3600"]
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 1Gi

The field serviceName: db points to the headless Service. Without it, there are no per-Pod DNS names.

The field selector.matchLabels must match template.metadata.labels. If they diverge, apply fails with an error:

The StatefulSet "bad" is invalid: spec.template.metadata.labels: Invalid value: {"app":"other"}: `selector` does not match template `labels`

The name in volumeMounts must match the template's name in volumeClaimTemplates. The server catches a typo there too, right at apply:

The StatefulSet "bad2" is invalid: spec.template.spec.containers[0].volumeMounts[0].name: Not found: "storage"

Deleting a Pod and deleting the StatefulSet

Delete Pod db-0, and the controller creates it again with the same name. PVC data-db-0 stays a separate object, and the new db-0 mounts the same disk again. The data is intact.

Delete the whole StatefulSet with kubectl delete sts db, and the PVCs stay by default. No data is lost. To remove the volumes, delete the PVCs separately.

This behavior is configured through the field spec.persistentVolumeClaimRetentionPolicy (stable since Kubernetes 1.32). It has two keys, whenDeleted and whenScaled, both with values Retain (the default) or Delete.

• whenDeleted decides what happens to PVCs when the StatefulSet is deleted.
• whenScaled decides what happens to the PVCs of the extra Pods when replicas shrinks. With Delete, shrinking from 3 to 2 takes data-db-2 along with Pod db-2.

The policy takes effect at the moment of deletion or scaling. If you shrink the replicas first and set the policy afterward, the departed Pod's PVC stays behind: you'll have to delete it by hand.

Useful commands

kubectl get sts

List of StatefulSets. sts is the resource's short name.

kubectl describe sts db

Details: the template, volumeClaimTemplates, and the events of Pod and PVC creation.

kubectl exec db-1 -- nslookup db-0.db.default.svc.cluster.local

Checks a neighbor's DNS name from inside a Pod. Cluster DNS is only visible from inside the cluster. nslookup from your laptop won't find this name. If the app's image doesn't have nslookup, a throwaway Pod works: kubectl run dns-test --rm -i --restart=Never --image=busybox:1.36.1 -- nslookup db-0.db.default.svc.cluster.local.

kubectl exec db-1 -- nslookup db.default.svc.cluster.local

The Service's own name. A headless one answers with the addresses of all ready Pods, a regular one with a single virtual IP.

kubectl get pvc

PVCs created from the template, and their status (Bound means the disk is provisioned).

Common mistakes

• A regular Service instead of a headless one. The Service's name hands out a single virtual IP, and replicas never learn each other's addresses. clusterIP can't be changed on an existing Service (may not change once set), so you recreate the Service.
• serviceName points to a Service with a different name. The Pods work, but they have no DNS names. The serviceName field itself also can't be changed once the StatefulSet exists.
• Expecting that deleting the StatefulSet frees up disks. PVCs stay until you delete them by hand or set whenDeleted: Delete.
• Deleting a PVC "for tidiness" while the StatefulSet still needs it. The next Pod with that number gets a new, empty disk.
• Checking a Pod's name with nslookup from your laptop. Cluster DNS is only visible from inside the cluster.
• Giving busybox's nslookup the short name db-0.db. It doesn't fill in domains from search and goes straight to an external DNS. The full name db-0.db.default.svc.cluster.local works.

What you'll need for the tasks

• kubectl get service and the CLUSTER-IP column.
• kubectl exec <pod> -- nslookup <full name> to check DNS from inside the cluster.
• kubectl get pvc and the PVC name formula.
• kubectl explain statefulset.spec.persistentVolumeClaimRetentionPolicy.
• kubectl patch and kubectl scale for StatefulSet.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your computer, instructions in its README.md.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-statefulset     # set up the starting state
./run.sh verify k8s-statefulset    # check the current step

In short

• StatefulSet gives stable names, ordered startup, a personal PVC, and a DNS name for each Pod.
• Pod DNS names only show up with a headless Service (clusterIP: None) and a serviceName pointing to it.
• PVC names are built as <template>-<sts>-<ordinal>, for example data-db-0.
• Deleting a Pod doesn't touch its PVC: a new Pod with the same name gets the same disk.
• Deleting a StatefulSet keeps PVCs by default; that behavior is changed by persistentVolumeClaimRetentionPolicy.