ReplicaSet: keeping N replicas

A controller that restores the needed number of Pod copies on its own, and why it's almost never written by hand.

Why you need a ReplicaSet

A Pod is the smallest unit Kubernetes runs, one copy of an application. If the node goes down or the process is killed by the OOM killer (a Linux kernel mechanism that stops processes when memory runs low), a Pod without a controller simply stops existing. Nobody brings it back. That's fine for a one-off task, but a web service needs something that keeps the right number of copies running.

A ReplicaSet is a controller that keeps exactly N copies of a Pod running. A controller here is a process that constantly checks reality against the plan. If there are fewer Pods than needed, the ReplicaSet creates new ones from the template. If there are more, it deletes the extra ones. It doesn't matter who deleted a Pod: a person by hand or a node that went down. The ReplicaSet brings up a replacement right away.

Manifest fields

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: web-rs
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27.3-alpine
          ports:
            - containerPort: 80

apiVersion: apps/v1: a ReplicaSet lives in the apps group, not in core. If you write plain v1, kubectl answers like this:

error: resource mapping not found for name: "web-rs" namespace: "" from "web-rs.yaml": no matches for kind "ReplicaSet" in version "v1"
ensure CRDs are installed first

The CRD hint here is a red herring: the real issue is apiVersion. The same applies to Deployment and StatefulSet.

spec.replicas: how many copies to keep. You can set it to 0: then the controller removes all the Pods, but the object itself stays. Handy for a temporary shutdown.

spec.selector: how the controller finds its Pods. matchLabels is a set of key-value pairs. All conditions are combined with AND. A label is a key-value tag on a resource. The entry app: web means "treat every Pod with the label app=web in this namespace as mine".

The selector must match the labels in template.metadata.labels. If the selector has app: web and the template has app: webapp, apply fails:

The ReplicaSet "web-rs" is invalid: spec.template.metadata.labels: Invalid value: {"app":"webapp"}: `selector` does not match template `labels`

The controller creates Pods from the template, then looks for them by the selector and has to find them. You can't change the selector of an existing ReplicaSet: the API answers such an edit with field is immutable. You need a different selector, which means you need a new ReplicaSet.

spec.template: the Pod template the controller uses to create copies. Inside is an ordinary Pod spec, without apiVersion and kind. The controller comes up with the Pod name itself: the ReplicaSet's name plus a random suffix, for example web-rs-jmtmd.

Whose Pods are these

Every Pod a ReplicaSet manages carries a reference to its owner in the metadata.ownerReferences field:

kubectl get pod web-rs-jmtmd -o jsonpath='{.metadata.ownerReferences}'

[{"apiVersion":"apps/v1","blockOwnerDeletion":true,"controller":true,"kind":"ReplicaSet","name":"web-rs","uid":"7ed57918-..."}]

This reference shows who's responsible for the Pod. Delete the ReplicaSet with kubectl delete rs web-rs, and its Pods get deleted along with it.

There's a subtlety. A ReplicaSet treats every Pod with matching labels as its own, even one it didn't create. The controller adopts such an unowned Pod: it sets its ownerReferences and counts it toward spec.replicas.

If a Pod labeled app: web was already running before the ReplicaSet appeared, the controller adopts it and creates one fewer copy than planned. If such a Pod shows up later, there will be more replicas than planned, and the controller deletes the extra one. It doesn't delete a random one: first Pods that aren't scheduled or ready yet, then the newest ones. That's why the Pod that disappears is usually the one you just created by hand.

What kubectl shows you

kubectl get rs web-rs

NAME     DESIRED   CURRENT   READY   AGE
web-rs   3         3         3       12s

• DESIRED: the value of spec.replicas, how many Pods are ordered.
• CURRENT: how many Pods actually exist.
• READY: how many of them are ready to take traffic.

If the three numbers match, the ReplicaSet is meeting the plan. If READY is less than DESIRED, look for the cause with kubectl describe rs web-rs and kubectl get pod. Keep in mind the numbers count every Pod under the selector, including outside ones the controller adopted.

At the end of kubectl describe rs web-rs's output, the Events block shows what the controller has been doing:

Events:
  Type    Reason            Age   From                   Message
  ----    ------            ----  ----                   -------
  Normal  SuccessfulCreate  11s   replicaset-controller  Created pod: web-rs-rdhzf
  Normal  SuccessfulCreate  11s   replicaset-controller  Created pod: web-rs-6l4dr
  Normal  SuccessfulCreate  11s   replicaset-controller  Created pod: web-rs-jmtmd
  Normal  SuccessfulCreate  8s    replicaset-controller  Created pod: web-rs-87zss

What self-healing looks like

Delete one Pod by hand and watch the list right away:

kubectl get pod -l app=web
kubectl delete pod web-rs-6l4dr
kubectl get pod -l app=web -w

Use a name from your own output instead of web-rs-6l4dr. In a couple of seconds, a new Pod shows up with a different random suffix in its name. That's the ReplicaSet at work. -w keeps the output open, exit with Ctrl+C.

You can change the replica count on the fly, without editing YAML:

kubectl scale rs/web-rs --replicas=5

If you then apply the old file with replicas: 3, the count goes back to three: kubectl apply brings the object in line with what's written in the file.

ReplicaSet and Deployment

In practice, a ReplicaSet is rarely written into manifests directly. On top of it sits a Deployment: a controller that can update the image version without downtime (a rolling update) and roll it back. But a Deployment creates a ReplicaSet under the hood anyway, so you need to understand its fields.

Common mistakes

• apiVersion: v1 instead of apps/v1: no matches for kind "ReplicaSet" in version "v1".
• The template's labels don't match the selector: `selector` does not match template `labels`. Compare both lines character by character, keys are case-sensitive.
• Trying to change the selector of a live ReplicaSet: field is immutable.
• A debugging Pod with the same labels as the replicas. The controller adopts it or deletes it as an extra, and kubectl get rs still shows everything is fine.
• Using kubectl scale instead of editing the file, then applying the old file. The replica count quietly rolls back.

What you'll need for the tasks

• kubectl apply -f <file> and a ReplicaSet manifest: apiVersion, selector, template.
• kubectl get rs and kubectl describe rs <name>, the Events block.
• kubectl get pod -l <selector> and kubectl get pod -L <keys> to see labels as columns.
• kubectl get pod <name> -o jsonpath='{.metadata.ownerReferences}' to find a Pod's owner.
• kubectl scale rs/<name> --replicas=<N>.
• Label commands from the lesson "Labels and selectors".

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your computer, instructions are in its README.md.

Homework

Practice the theory on your own cluster: 3 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-replicaset     # set up the starting state
./run.sh verify k8s-replicaset    # check the current step

In short

• A ReplicaSet keeps exactly spec.replicas copies of a Pod and restores deleted ones.
• apiVersion is only apps/v1, otherwise kubectl won't find that kind.
• selector.matchLabels has to match template.metadata.labels, and the selector itself can't be changed after creation.
• The selector catches any matching Pod in the namespace, even one created by hand. A Pod's owner is shown by ownerReferences.
• In real manifests you write a Deployment, and it creates the ReplicaSet itself.