Canary deployment

The traffic share through a replica ratio: how it works on a plain Service, where the precision falls short, and when it's time to reach for a service mesh.

Why you need a canary

Canary is a deploy on a growing share of traffic. The new version rolls out first to a small percentage (5-10%), you watch metrics, errors, p99 latency (the response time that 99% of requests fall under). If everything looks good, you raise the share to 25%, 50%, 100%. If something breaks, the rollback is quick: only the canary's share of users was affected.

The name comes from mines. Miners carried a caged canary down with them. The bird stopped singing when there was a gas leak, and that was the signal to leave. In a deploy it's the same idea: a small slice of load catches the problem before the rest does.

Concepts

Pod - the smallest unit of execution, a container runs inside it. Deployment - an object that keeps a target number of identical Pods running. Service - an object that routes traffic to a set of Pods by labels. selector - a set of labels a Service uses to pick Pods: a Pod matches the selector only if it carries every key=value pair.

EndpointSlice - an object listing the Pod addresses a Service sends traffic to. Kubernetes fills it in automatically from the Service's selector. kube-proxy (a component on every node) turns this list into network rules.

Service mesh - a separate traffic-management layer on top of the cluster. In this lesson it's only here for comparison.

How it works

Without a service mesh, canary is done through a replica ratio. One Service catches all of the app's Pods, both stable and canary. kube-proxy spreads new connections roughly evenly across the addresses in the EndpointSlice. To give the canary about 40%, you keep 2 canary Pods for 3 stable ones.

Service app   selector: app=app
  stable  ●●●   3 of 5  ~60%
  canary  ●●    2 of 5  ~40%

Minimal setup

You need two Deployments and one Service. In the example both versions reply with a string naming their own version, so you can see which Pod answered. The image is the same for both (busybox:1.36.1, a built-in HTTP server), the version is set through the APP_VERSION variable. In real life a canary carries a different image tag.

Stable:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-stable
spec:
  replicas: 3
  selector:
    matchLabels: {app: app, version: stable}
  template:
    metadata:
      labels: {app: app, version: stable}
    spec:
      containers:
        - name: app
          image: busybox:1.36.1
          env:
            - name: APP_VERSION
              value: v1
          command: ["sh", "-c", "mkdir -p /tmp/www && echo \"app $APP_VERSION $(hostname)\" > /tmp/www/index.html && exec httpd -f -p 8080 -h /tmp/www"]
          ports:
            - containerPort: 8080

Canary, one replica with the new version. It differs by name, the version label's value, and the version itself:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-canary
spec:
  replicas: 1
  selector:
    matchLabels: {app: app, version: canary}
  template:
    metadata:
      labels: {app: app, version: canary}
    spec:
      containers:
        - name: app
          image: busybox:1.36.1
          env:
            - name: APP_VERSION
              value: v2
          command: ["sh", "-c", "mkdir -p /tmp/www && echo \"app $APP_VERSION $(hostname)\" > /tmp/www/index.html && exec httpd -f -p 8080 -h /tmp/www"]
          ports:
            - containerPort: 8080

The version label in each Deployment's selector is there so the two Deployments' selectors don't overlap. The Kubernetes documentation warns directly: Deployments with overlapping selectors can behave unexpectedly.

The Service catches both Deployments through a shared label:

apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  selector:
    app: app           # no version, so it catches both versions
  ports:
    - port: 80
      targetPort: 8080

You can see who answered from inside the cluster. Every run of wget opens a new connection, so each request picks a Pod again:

kubectl exec deploy/app-stable -- sh -c 'for i in $(seq 1 100); do wget -qO- -T 2 http://app; done' | cut -d" " -f2 | sort | uniq -c

     79 v1
     21 v2

With 3 stable and 1 canary the calculated share is 25%. Your number will differ, but it'll be close to it.

How the ratio is shifted

The command kubectl scale changes the replica count:

kubectl scale deploy app-canary --replicas=2

Now the canary is 2 of 5, about 40%. There's no separate "traffic share" object: only the number of Ready Pods of each version sets the ratio.

Once the share is clean and the metrics look fine, the release gets closed. There are two ways. First: scale the canary up to full size and the stable down to zero, and the canary becomes the new stable. Second: roll out the new version to stable through a normal rolling update, and scale the canary down or delete it. The second way is more convenient when the name app-stable is hardcoded in CI, dashboards, and alerts.

Granularity and precision

The traffic share equals the share of Ready Pods. To give 1%, you need 1 Pod out of 99. To give 0.1%, you need 1 out of 999. On small setups the step is coarse: 1 out of 4 is already 20%.

The balancing isn't perfect. kube-proxy picks a Pod for every new TCP connection, not for every request. A client with keep-alive holds one connection and sends all its requests to a single Pod. On low traffic, 90/10 easily drifts to 85/15. It takes thousands of connections for the statistics to even out.

There's also a kube-proxy nuance. In iptables mode (rules in the Linux kernel), it doesn't cycle through addresses: every new connection walks the rule chain -m statistic --mode random and gets a random address with equal probability. IPVS mode does cycle through. In practice there's no difference: there are no weights either way, over a long run the shares come out equal, so the replica ratio is the traffic ratio.

When you need a service mesh

Without a service mesh, control comes down to the number of Pods. You can't route 10% of users to the canary by userID. You can't send only requests with the header x-canary: true to the canary. Dedicated tools can do this:

• Istio through a VirtualService with route weights;
• Linkerd through the Gateway API: the split is described with weights in HTTPRoute.backendRefs, the old SMI TrafficSplit is deprecated;
• Argo Rollouts with built-in canary logic;
• Flagger to automate canary on top of Istio or Linkerd.

ingress-nginx's canary annotations (nginx.ingress.kubernetes.io/canary) used to be on this list. The project's support ended in March 2026, so this path is closed on new clusters.

All of them cost more to run than kubectl scale. That's why small projects start with a canary at the Pod level.

Related commands

kubectl scale deploy app-canary --replicas=2
kubectl get pods -l app=app --show-labels
kubectl get endpointslice -l kubernetes.io/service-name=app
kubectl rollout status deploy/app-canary
kubectl logs -l app=app,version=canary --tail=20

The old v1 Endpoints object is deprecated since 1.33, and kubectl get endpoints app prints a warning:

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice

So check the list of addresses through EndpointSlice, as in the block above.

Common mistakes

• The Service's selector has version. Then it catches only stable or only canary. A Service's selector should be shared, by app.
• Canary Pods don't show up in the EndpointSlice. Causes: the Pod isn't Ready, the template labels don't match the Service's selector, a failing readinessProbe. Check kubectl get pods --show-labels and compare it with the Service's selector pair by pair.
• A Deployment's selector can't be changed: the API answers field is immutable. If the canary was created with the wrong labels in its selector, delete it and create it again.
• The two Deployments' selectors overlap (both have only app=app). The version label is needed in both the selector and the Pod template.
• The stable and canary metrics aren't tagged. They'll mix together in Prometheus. Add version to the metric labels.
• The canary never gets closed: the old and new versions live together for weeks.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl get pods --show-labels, kubectl get svc <svc> -o jsonpath='{.spec.selector}', and kubectl get endpointslice -l kubernetes.io/service-name=<svc>.
• kubectl get deploy <name> -o yaml and kubectl apply -f <file>, to recreate a Deployment.
• kubectl scale deploy <name> --replicas=<N> and counting shares: the canary's share equals canary / (canary + stable).
• A wget loop from a Pod, like in the "Minimal setup" section, to measure the share.
• For the mini-project: Job, CronJob, the RollingUpdate strategy, kubectl rollout history from earlier lessons in the module.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-canary     # set up the starting state
./run.sh verify k8s-canary    # check the current step

In short

• The canary's traffic share equals its share of Ready Pods among all Pods behind the Service.
• A Service's selector takes only the shared app label, without version. A Deployment's selector, on the other hand, includes version.
• The ratio is moved through kubectl scale, there's no separate object for it.
• The precision is coarse: kube-proxy picks a Pod per connection, on low traffic the real distribution drifts away from the calculated one.
• Splitting by headers, userID, and exact weights requires Istio, Linkerd, Argo Rollouts, or Flagger.
• A finished canary gets closed: the new version on all Pods, the canary scaled to zero or deleted.