Readiness, liveness, and startup probes

Why this matters

The container is running and the main process is alive. That doesn't mean the application is working. It might be warming a cache, and then the first requests get an error. Or the process has hung: it's sitting in memory and not responding. Kubernetes doesn't see any of this on its own until you tell it how to check. A wrong check hurts more than having none: one bad address in a livenessProbe can restart every Pod of an application in a loop.

What a probe is

kubelet is the Kubernetes component that runs containers on a node and watches their state. A probe is a check that kubelet runs for a container on a regular basis. Based on the result, kubelet decides whether the container is ready for traffic and whether it's time to restart it.

There are three kinds of probe, and each one answers its own question.

Probe	What question it answers	What happens on failure
readinessProbe	is the container ready for traffic	the Pod is marked not ready, the Service stops sending it requests
livenessProbe	is the process genuinely alive	kubelet restarts the container
startupProbe	has startup finished	until it passes, the other two don't run; failing past the threshold restarts the container

readinessProbe

A Service is an object that distributes network traffic across a group of Pods. It only picks ready Pods. Until readinessProbe returns success, the Pod's READY column shows 0/1, and in the Service's EndpointSlice its address is marked as not ready (Ready: false in kubectl describe endpointslice). No requests go there, and the Endpoints: line in kubectl describe svc doesn't show that address.

readinessProbe runs for the whole life of the container, not just at startup. If the application loses its database and can't serve requests, readiness starts failing, and the Pod temporarily drops out of the Service. The container itself doesn't restart. Once the check passes again, the Pod returns to the Service.

livenessProbe

livenessProbe answers the question "is the process really alive". If it fails failureThreshold times in a row (3 by default), kubelet kills the container and starts it again. In kubectl get pods output, RESTARTS goes up, and after several restarts in a row the Pod goes into CrashLoopBackOff: kubelet pauses between attempts, and the pauses grow.

livenessProbe helps against hangs: the process exists, but requests aren't being handled. A restart fixes that. But a restart doesn't fix a database that's down or a neighboring service that's unreachable. That's why liveness checks only the process itself and doesn't call out to external dependencies.

startupProbe

Some applications take a long time to start: cache warmup, migrations, loading a model. If liveness starts checking such an application before it has opened its port, kubelet restarts it partway through startup. And so on in a loop.

startupProbe solves this. Until it passes, kubelet runs neither liveness nor readiness. The time allowed for startup is the product of periodSeconds and failureThreshold:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 30

Here the application has up to 300 seconds to start. If the check never passes in that time, kubelet restarts the container. Once startupProbe has passed once, it never runs again, and from then on liveness and readiness work with their usual intervals.

If the application starts in a couple of seconds, you don't need a startupProbe.

How a probe check works

The way a check is made is set by one of four fields:

• httpGet: kubelet makes an HTTP request to the container's path and port. A response code from 200 to 399 counts as success, anything else as failure.
• tcpSocket: kubelet tries to open a TCP connection to the port. If it opens, that's a success.
• exec: kubelet runs a command inside the container. Exit code 0 counts as success.
• grpc: the standard gRPC health check, if the application implements it.

The rest of the fields set the rhythm:

Field	What it sets	Default
initialDelaySeconds	pause after container start before the first check	0
periodSeconds	interval between checks	10
timeoutSeconds	how long to wait for a response	1
failureThreshold	how many failures in a row count as a probe failure	3
successThreshold	how many successes in a row are needed after a failure (always 1 for liveness and startup)	1

A minimal example for a web application on port 80:

readinessProbe:
  httpGet:
    path: /
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10

Probes are described inside the container, next to name and image. In a Deployment that's spec.template.spec.containers[]. You can't change the probes on a Pod that's already running: you edit the template in the Deployment, and it creates new Pods.

The time until a hung container restarts is calculated as periodSeconds × failureThreshold. With the default settings that's about 30 seconds. If liveness checks a container that hasn't opened its port yet, the first check happens after initialDelaySeconds, and the restart happens after roughly initialDelaySeconds + periodSeconds × (failureThreshold - 1).

camelCase in keys

Keys in Kubernetes manifests are written in camelCase. httpGet is correct, httpget is not. A typo in casing gets an error from the API server, and the object isn't created (strict field validation is on by default):

Error from server (BadRequest): error when creating "bad.yaml": Pod in version "v1" cannot be handled as a Pod: strict decoding error: unknown field "spec.containers[0].readinessProbe.httpget"

How to check probes and their results

Field descriptions straight from the cluster:

kubectl explain pod.spec.containers.readinessProbe
kubectl explain pod.spec.containers.livenessProbe.httpGet

Which probes are set on Deployment app:

kubectl get deploy app -o yaml | grep -A8 Probe

How they fire. kubectl describe pod has Liveness:, Readiness:, and Startup: lines with the settings, and the Events block shows the failures:

Warning  Unhealthy  ...  Liveness probe failed: Get "http://10.244.0.5:8080/healthz": dial tcp 10.244.0.5:8080: connect: connection refused
Normal   Killing    ...  Container app failed liveness probe, will be restarted

The line Killing ... failed liveness probe means liveness caused the restart, not a crash of the process itself. See what the application managed to write before the restart with kubectl logs <pod> --previous (the lesson on kubectl logs).

Common mistakes

• liveness checks an endpoint that calls the database. The database went down, and every Pod of the application restarted with it. A restart wouldn't have brought the database back anyway.
• liveness with no startupProbe on a slow-starting application. The container is killed before it opens its port, and the Pod goes into CrashLoopBackOff.
• initialDelaySeconds: 0 on an application that doesn't start instantly. The first checks knock on a closed port.
• The probe looks at the wrong port or path. Readiness never passes, the Pod sits at 0/1, the Service has no ready addresses.
• Readiness and liveness with the same settings on the same "deep" endpoint. Readiness should know about dependencies, liveness shouldn't.
• Fixing restarts by removing livenessProbe. The restarts go away, but nobody restarts a hung process anymore either.

What you'll need for the tasks

• kubectl get pods: the READY and RESTARTS columns.
• kubectl describe pod: the Liveness:, Readiness:, and Startup: lines, and the Unhealthy, Killing events in the Events block.
• kubectl logs with -c and --previous (the lesson on logs).
• Where probes live in a Deployment and how to fix them: kubectl edit deploy or kubectl apply of the edited manifest.
• kubectl describe svc <svc>: the Endpoints: line only shows ready addresses. kubectl get endpointslices shows the unready ones too, readiness is visible in kubectl describe endpointslice (Ready: false).
• The formula periodSeconds × failureThreshold.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-readiness-liveness     # set up the starting state
./run.sh verify k8s-readiness-liveness    # check the current step

In short

• readinessProbe decides whether to let Service traffic into the Pod. A failure removes the Pod from the Service without a restart.
• livenessProbe decides whether to restart the container. It checks only the process itself and doesn't depend on external services.
• startupProbe holds the other two back while a slow application starts. Time for startup: periodSeconds × failureThreshold.
• httpGet treats codes 200-399 as success. By default periodSeconds is 10, timeoutSeconds is 1, failureThreshold is 3.
• Probes live inside the container in the Deployment template, keys are strictly camelCase.