Health and readiness endpoints

Why this matters

A load balancer or an orchestrator can't read an app's mind. It needs a way to ask "are you okay?" over the network. So the app exposes a special HTTP endpoint that quickly answers about its own state. By convention it's called /healthz or /health. Kubernetes itself keeps separate /livez and /readyz on the API server, where /healthz is considered deprecated.

There are three probes, and each answers its own question: has the app started, is it alive, and is it ready to take traffic. Below they're described in Kubernetes terms, where each one has its own manifest field: startupProbe, livenessProbe, readinessProbe.

How the orchestrator asks the question

kubelet on the node runs the probe. For an HTTP probe it sends a GET to the given path and port on the container. Any code from 200 to 399 counts as success, everything else and a timeout count as failure.

curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8080/healthz

200

How often to ask and how many failures to tolerate are set by the probe's fields. Defaults:

Field	Default	What it means
initialDelaySeconds	0	how long to wait after the container starts before the first probe
periodSeconds	10	how often to ask
timeoutSeconds	1	how long to wait for a response
failureThreshold	3	how many failures in a row count as a final failure

So with default settings a failure is confirmed after about 30 seconds: three failures 10 seconds apart. Besides HTTP, a probe can also be tcpSocket (could the port be opened), exec (a command's exit code inside the container), and grpc.

Liveness: is the app alive

Liveness answers the question "has the app hung?". If liveness fails, kubelet considers the container dead and restarts it. This fixes hangs the process can't get out of on its own: a deadlock, a leak, an infinite loop.

Liveness is made as simple as possible: return 200 if the process is alive and responding. It shouldn't call a database or other external services. A restart doesn't fix someone else's database, and a temporary database outage would trigger a wave of needless restarts. While replicas are restarting, they serve nobody at all, not even requests that don't need the database.

Readiness: the app is ready for traffic

Readiness answers a different question: "can I send requests here yet?". The app has started, but it's still warming up a cache, establishing a database connection, reading config. At that moment it's alive, but not ready. While readiness is failing, the Pod isn't Ready, and the Service doesn't send it traffic: the Pod's address in the EndpointSlice is marked not ready.

Unlike liveness, readiness is exactly where you check dependencies: is there a database connection, is the queue available. No database means not ready to serve, and traffic goes to healthy replicas instead. A readiness failure doesn't restart the container. The probe keeps running, and as soon as the dependency is back, the replica gets traffic again.

Example setup of both probes on a container:

livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

Here /livez answers 200 as long as the process responds to requests, while /readyz also checks the database connection.

Startup: the app is still starting

The third probe solves the slow-start problem. Say an app needs two minutes to warm up: bring up the JVM, read a large config, run migrations. Liveness with the usual settings would decide within that time that the process has hung, and restart it. The app starts over, fails to make it again, and so on in a loop. The Pod ends up in CrashLoopBackOff for no real reason: kubelet restarts the container with a growing backoff, up to 5 minutes.

This used to be patched around: give liveness a huge initialDelaySeconds, or allow dozens of failures. The side effect isn't nice: after warmup the probe becomes sluggish, and a real hang gets caught not in fifteen seconds but in five minutes.

Startup probe separates the two modes. It answers the question "has the app started" and only runs during container startup. It landed in Kubernetes in 1.16 and became stable in 1.20.

Here's the order. Until startup returns success, liveness and readiness don't run at all. kubelet just waits for as long as the startup probe's budget allows. Once startup passes, it stops running for good, and liveness and readiness kick in with their normal, short settings. If the container restarts later, everything starts over from startup.

container starts
   │
   ├── startup: polled until it answers success
   │       (liveness and readiness wait)
   │
   └── startup passed → doesn't run again for the rest of the container's life
           │
           ├── liveness   — until the container stops
           └── readiness  — until the container stops

The startup budget is failureThreshold × periodSeconds. Thirty attempts 10 seconds apart give five minutes to start. Miss that, and kubelet restarts the container.

startupProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  failureThreshold: 30

A separate endpoint for startup usually isn't needed: most of the time it watches the same path as liveness, just with a much more generous retry limit.

Why separate them

The split produces correct behavior in three cases.

• A long start. Startup keeps liveness and readiness off while the app is coming up, and stops restarts from looping.
• Warmup after starting. The app has already started, but it's still filling a cache: liveness succeeds (don't restart), readiness doesn't yet (don't send traffic). Without readiness, the load balancer would flood an unwarmed replica with requests.
• A temporary dependency outage. The database blips. Readiness goes to "not ready", traffic moves to other replicas, the replica isn't restarted. Once the database is back, readiness succeeds again.

If liveness called the database, an outage like that would trigger a cascade of restarts for no reason.

Connection to rollouts

Readiness ties back to earlier lessons. Rolling update and canary rely on it: the Deployment counts a new replica as available only once its Pod has become Ready. Until new replicas are Ready, the rollout doesn't proceed and old replicas aren't removed.

The connection to graceful shutdown works like this. When a Pod is deleted, Kubernetes itself marks its address in the EndpointSlice as not ready. A separate readiness check isn't needed for that. But an external load balancer that only knows about a replica through its health endpoint has to be told "not ready" by the app itself on SIGTERM: it starts answering readiness with an error and finishes the requests it already has in flight.

Common mistakes

• Liveness checks dependencies. The database is unavailable, every replica restarts in a loop and serves nobody. Keep database checks in readiness.
• Liveness and readiness on the same path that hits the database. The same mistake, just harder to spot: the path is called /health, but inside it queries the database.
• A slow start is patched with initialDelaySeconds on liveness. Startup passes, but a later hang is caught only after minutes. That's what startup is for.
• No readiness at all. A Pod without readiness counts as ready right after the container starts, and traffic hits an unwarmed replica.
• A heavy probe. The endpoint runs a complex query and doesn't fit within timeoutSeconds (1 second by default). Under load, the probe fails on its own.

What to remember

Liveness answers "is it alive" (failure → restart) and is kept simple, with no dependencies. Readiness answers "is it ready for traffic" (failure → pull out of rotation) and checks the database and queues. Startup answers "has it started": until it passes, liveness and readiness don't run, so a slow start doesn't turn into an endless restart.

Next lesson: 12-factor configuration, how to keep settings in the environment.

What you'll need for the tasks

• The lab's kind cluster and export KUBECONFIG="$PWD/setup/kubeconfig".
• kubectl get pods (the READY and RESTARTS columns), kubectl describe pod (the Liveness, Readiness, Startup fields and the Events section), kubectl logs --previous.
• kubectl logs deploy/<name> --since=60s and grep, to check the client's log for the last minute.
• kubectl scale deployment/<name> --replicas=N, kubectl edit or kubectl apply -f, kubectl rollout restart, kubectl rollout status.
• Probe fields: httpGet.path, periodSeconds, failureThreshold.