Blue-Green deployment

Two environments and one Service switch: how to get an instant rollback, and what it costs you.

Why

Rolling update changes Pods one at a time, and rollback goes just as gradually: the old Pods have to be brought back up again. If a new version breaks in production, minutes of rollback are expensive. Blue-Green solves this differently: the old version stays up until the new one has proven itself, and sending traffic back to it is a single command.

What Blue-Green is

Blue-Green is a zero-downtime deployment strategy with an instant rollback. Production holds two independent environments. One is called blue, and it takes all the traffic. The other is called green, deployed and ready, but not getting any traffic.

When green is ready, traffic switches to it. If something breaks, traffic switches back to blue. Blue stays alive the whole time, so rollback is instant.

In Kubernetes, the Service plays the role of the switch. A Service is an object that sends traffic to a set of Pods chosen by labels. A label is a key-value pair on an object. A selector is a set of labels the Service uses to pick Pods.

Change the selector from color=blue to color=green, and the Service's address list is rebuilt. That list lives in EndpointSlice objects: they hold the IP addresses of the Pods the Service sends traffic to. kube-proxy (a component on every node that keeps the network rules for Services) updates its rules, and new requests go to green. This usually takes seconds.

The main benefit is that rollback runs at the same speed as the rollout. One patch sends the selector back. What you pay for this is resources: during the switch window, both versions are alive in the cluster at once. A Pod using 500 MB of RAM at 5 replicas means an extra 2.5 GB. For critical services, that safety net is worth it. For small microservices with frequent releases, the extra resource cost outweighs the benefit.

The minimal setup

You need two Deployments with different values of the color label. A Deployment is an object that keeps a set number of identical Pods running.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-blue
spec:
  replicas: 3
  selector:
    matchLabels: {app: app, color: blue}
  template:
    metadata:
      labels: {app: app, color: blue}
    spec:
      containers:
        - name: app
          image: nginxinc/nginx-unprivileged:1.27.3-alpine
          ports:
            - containerPort: 8080

The second Deployment is the same, but named app-green, with color: green and image nginxinc/nginx-unprivileged:1.27.4-alpine.

One Service with an explicit color in its selector:

apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  selector:
    app: app
    color: blue       # switch blue/green here
  ports:
    - port: 80
      targetPort: 8080

How traffic gets switched

First make sure green is fully ready:

kubectl rollout status deploy/app-green

Then switch:

kubectl patch svc app \
  -p '{"spec":{"selector":{"app":"app","color":"green"}}}'

kubectl patch writes the new color into spec.selector. The EndpointSlice controller notices the change, scans the Pods in the namespace, and adds the ones that match the new selector and are ready into an EndpointSlice. kube-proxy on every node watches the EndpointSlice and updates its network rules.

A patch merges maps by default: {"spec":{"selector":{"color":"green"}}} changes only color, and app stays. The command kubectl set selector svc app color=green behaves differently: it replaces the whole selector, and app disappears from it.

Connections opened before the switch live out their life on blue. New connections go to green.

Rollback

kubectl patch svc app \
  -p '{"spec":{"selector":{"app":"app","color":"blue"}}}'

The blue Deployment has been alive the whole time, its Pods are ready. Rollback takes the same seconds as the switch: there are no new Pods to wait for.

Related commands

kubectl get deploy
kubectl get svc app -o jsonpath='{.spec.selector}'
kubectl get endpointslice -l kubernetes.io/service-name=app
kubectl get pods -l color=green -o wide
kubectl rollout status deploy/app-green

The address list used to be checked with kubectl get endpoints app. The v1 Endpoints object is deprecated as of Kubernetes 1.33, and the command prints a warning:

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice

That's why EndpointSlice is used above instead: every such object carries a kubernetes.io/service-name label with the Service's name.

What commonly breaks

• Green isn't Ready yet, but the Service is already switched. The Service is left with no ready addresses at all, and requests inside the cluster get Connection refused. Run kubectl rollout status to completion first, then patch.
• The Service selector has no color key. The Service picks up Pods of both versions, and some users see the old one.
• The Deployment's spec.selector.matchLabels is missing a unique color. The two Deployments' selectors overlap. While both are alive, each one manages only its own ReplicaSets (linked by ownerReferences). But delete blue with --cascade=orphan (done to recreate a Deployment without downtime), and green will adopt blue's ReplicaSets as its own and scale them down to zero: blue's Pods disappear. The Kubernetes documentation explicitly forbids overlapping selectors on Deployments. Keep color in both matchLabels and the template's labels.
• Blue was deleted after the switch. Rollback is no longer instant, you now have to wait for blue to roll out again. Keep the old color around at least until the next release.

When Blue-Green doesn't fit

• Stateful workloads with a single volume in RWO mode (ReadWriteOnce: the volume mounts for writing on only one node at a time). Two colors on different nodes can't hold such a volume at the same time. You need RWX (ReadWriteMany: writable from several nodes) or a different approach.
• Very expensive Pods, where doubling the cost doesn't pay off. Use rolling update here instead.
• You need a percentage of traffic going to the new version, say 10%. That's canary, not Blue-Green.

What you'll need for the tasks

• Pod labels: kubectl get pod -l app=<name> --show-labels.
• Service selector: kubectl get svc <name> -o jsonpath='{.spec.selector}'.
• Service addresses: kubectl get endpointslice -l kubernetes.io/service-name=<name>.
• Scaling and waiting for readiness: kubectl scale, kubectl rollout status.
• Switching the selector: kubectl patch svc from the "How traffic gets switched" section.

The labs live in the labs/ folder: each one brings up its own kind cluster on your machine, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-blue-green     # set up the starting state
./run.sh verify k8s-blue-green    # check the current step

In short

• Two Deployments with a color label, one Service decides who gets traffic.
• Switching and rollback are both a single kubectl patch on spec.selector.
• Blue can't be deleted right away: it's the safety net for an instant rollback.
• The Service selector must include color, or traffic goes to both versions.
• The cost of this approach is double the resources during the switch window.
• Blue-Green doesn't give you a percentage split of traffic. For that you need canary.