Deployment strategies: rolling, blue-green, canary

Why this matters

The simplest deploy: stop the old version and start the new one. While the new one starts, the service is down. If the new version is broken, users get errors, and a rollback turns into another deploy with the same downtime. Nobody deploys to production like that.

Three strategies solve this differently: rolling, blue-green and canary. All three keep the service available during the rollout and leave a way back.

Concepts

• Rollout: replacing a running version of the application with a new one.
• Rollback: reverting to the previous version if the new one misbehaves.
• Replica of the application: a single instance (container, Pod, process) that serves requests. There are usually several replicas, and a load balancer splits traffic between them.
• Load balancer: the entry point that decides which replica gets each request. This can be nginx, a cloud load balancer, or a Service in Kubernetes.

Rolling

A rolling update replaces replicas of the application one at a time. Say there are four replicas. You stop one old replica, start a new-version replica in its place, wait until it's ready, and move on to the next one. Kubernetes by default does it the other way around: it first brings up a new replica on top of the desired count, then stops an old one. The order is set by the maxSurge and maxUnavailable parameters, both 25% by default. Either way, the old and new versions run side by side for a while.

Pro: no downtime, no need for double the servers. Con: during the rollout both versions run in production at once, so they must be compatible, especially on the database schema. A rollback is done the same way, a gradual rolling update backward. In Kubernetes this is the Deployment's default strategy.

Blue-green

Blue-green keeps two full environments. Blue serves production, green stands next to it on the new version and gets no traffic. Once green is tested, the load balancer switches all traffic to it at once.

Pro: the switch is instant, and so is the rollback, you just need to point traffic back at blue. Con: for the rollout you need two full environments, meaning double the resources. This strategy suits releases where a fast rollback matters.

The switch itself happens at the load balancer level: you change where it sends requests and ask it to reload its settings. The entry point itself must keep running during this, otherwise the downtime just comes back another way.

Canary

Canary first sends a small share of traffic to the new version. Say 5% of requests go to v2, the rest to v1. You watch metrics and errors. If everything looks good, you raise the share: 25%, 50%, 100%. If something goes wrong, you drop the share to zero, and only 5% of users ever saw the problem.

Pro: a problem gets caught on a small share of traffic, so the blast radius is small. Con: you need a load balancer that can split traffic in a set proportion, and good metrics to tell whether the canary is healthy. The name comes from mining canaries: the bird reacted to dangerous gas before the miners did.

The share is usually set with server weights in the load balancer. Exactly how a weight turns into a percentage depends on the tool, so it's worth measuring the real share: send a hundred requests and count how many each version answered.

What to pick

• No spare resources, versions are compatible, you need a simple rollout with no downtime: rolling.
• An instant rollback matters, and you have resources for a second environment: blue-green.
• High risk, you have metrics and controlled traffic: canary.

Any of the three strategies requires the application to shut down cleanly when replaced and to report its readiness. See the app-patterns track lessons "Graceful shutdown" and "Health endpoints" for that.

Common mistakes

• Removing blue right after switching to green. An instant rollback is only possible while the old environment is still alive and ready to take traffic.
• Restarting the load balancer so it picks up the new config. While it's starting, requests fail, and a rollout without downtime stops being one. Most load balancers can reload settings on the fly: for nginx that's nginx -s reload.
• Shipping an incompatible database migration together with a rolling or canary rollout. The old version, still serving part of the traffic, starts failing on the new schema.
• Eyeballing the canary's share from the config. Verify by measuring: a series of requests and a count of each version's answers.
• Running a canary without metrics. If you can't see errors and latency broken out for v2, a small traffic share buys you nothing: users will find the problem anyway.

What you'll need for the tasks

• docker compose up -d, docker compose ps, docker compose logs, docker compose exec with the -f setup/docker-compose.yml flag;
• the upstream block in the nginx config: the list of servers the load balancer sends requests to;
• nginx -t (check the config) and nginx -s reload (reload without stopping), both inside the load balancer container;
• curl -s http://127.0.0.1:PORT/ and a bash loop to send a series of requests and count the answers:

for i in $(seq 1 20); do curl -s http://127.0.0.1:8080/; done | sort | uniq -c

The next lesson covers the rest of the pipeline patterns: gates, manual approvals and parallel branches.