Weekend CrashLoop: an aggressive livenessProbe

The alert comes in Saturday night: the service is in CrashLoopBackOff, the restart count keeps climbing. By Monday it's fixed itself, and the ticket gets closed as "unclear, couldn't reproduce." A week later it happens again.

The time-of-day pattern is the clue here. "Works fine under load, crashes at rest" usually means the livenessProbe doesn't survive a cold start.

Why "on weekends" is a diagnosis

On weekdays the service gets traffic. The JIT is warmed up, caches are full, database connections are established, the health check answers in 50ms. On weekends there's no load, but there is maintenance: rolling updates, evictions, node reboots. The Pod gets recreated, and the new container starts cold. The first response after start can take several seconds, and a full warmup can take tens of seconds.

If the livenessProbe fires every periodSeconds: 10 and tolerates failureThreshold: 3 misses, kubelet restarts the container about 30 seconds after the first failure. The new container starts just as slowly and gets killed before it warms up, like the one before. The loop closes.

The same symptom also shows up for other reasons, for example OOMKilled right after start. Telling one from the other using describe is covered in the lesson "livenessProbe restarts the container".

Which probe settings break everything

The defaults below come from the Kubernetes documentation (Configure Probes).

initialDelaySeconds

How long to wait after the container starts before the first check. Defaults to 0. If the app hasn't opened its port yet, the check gets connection refused and counts as a failure. A few seconds of delay is enough for an ordinary HTTP service. If startup takes a long time, don't stretch this delay, add a startupProbe instead (section below).

periodSeconds

The interval between checks. Defaults to 10, minimum 1. Going below 5 rarely makes sense: more load on the app, no extra useful signal.

timeoutSeconds

How long to wait for a response to one check. Defaults to 1. That's not enough for a heavy Java app, a loaded Python service, or an app with a cold cache: a slow but alive response counts as a failure. A sane range is 3-5.

failureThreshold

How many failures in a row it takes for kubelet to restart the container. Defaults to 3. At 1, any random delay kills the container. A sane middle ground is 3-5.

When kubelet will restart the container

The first check runs initialDelaySeconds after the container starts, then every periodSeconds after that. A restart happens after failureThreshold failures in a row. If the app doesn't respond from the very start, the upper bound is:

time to restart ≈ initialDelaySeconds + periodSeconds × failureThreshold

In practice it comes a bit sooner: the first failure already happens at the moment of initialDelaySeconds. After deciding to restart, kubelet sends the process SIGTERM and waits terminationGracePeriodSeconds (30 by default), then SIGKILL.

An aggressive check:

livenessProbe:
  httpGet: { path: /, port: 80 }
  initialDelaySeconds: 0
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 2

Time to restart: no more than 0 + 5 × 2 = 10 seconds. Any app that needs more than that to start is guaranteed to end up in CrashLoopBackOff. A 1-second timeout adds a second trap: even an open port won't save you if the first responses take longer than a second.

Safer:

livenessProbe:
  httpGet: { path: /healthz, port: 80 }
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3

Time to restart: up to 30 + 10 × 3 = 60 seconds. A start under half a minute survives with room to spare.

cold start          first response ~10 sec
aggressive probe     0 + 5×2  = 10 sec
safe probe          30 + 10×3 = 60 sec

What this looks like in diagnostics

kubectl get pod -l app=web

The RESTARTS column climbs, STATUS flips between Running and CrashLoopBackOff.

kubectl describe pod -l app=web

Look at three places. The Liveness: line shows the check settings in one line:

Liveness:  http-get http://:80/healthz delay=0s timeout=1s period=5s #success=1 #failure=2

The Last State block shows how the previous run ended, and Events shows who stopped it:

Warning  Unhealthy  kubelet  Liveness probe failed: Get "http://10.244.0.5:80/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal   Killing    kubelet  Container web failed liveness probe, will be restarted

The tail of the Unhealthy message says exactly what failed:

• connect: connection refused: the port isn't open yet, or the check is hitting the wrong port;
• context deadline exceeded: the port is open, but no response arrived within timeoutSeconds;
• HTTP probe failed with statuscode: 503: the app responded, but with a code outside 200-399.

If Last State: Terminated shows Exit Code: 137 next to it, the process got SIGKILL: it didn't exit within the grace period after SIGTERM. Together with a Killing event, that means kubelet restarted the app; it didn't crash on its own. Code 137 also happens with OOMKilled, so check Reason and the events, not just the code.

kubectl logs -l app=web --previous

The previous run's log gives the other half of the picture. A clean start with no errors that breaks off in the middle of initialization confirms the probe theory.

When it's not the probe

CrashLoopBackOff by itself says only one thing: the container keeps exiting, and kubelet restarts it with an ever-growing pause. Who ended the container shows up in the same three places in describe.

If there's no Liveness: line at all, or the events have no Killing ... failed liveness probe, kubelet never touched the container. The process exited on its own. Last State then shows Reason: Error with whatever code the app returned, most often Exit Code: 1. Reason: OOMKilled with code 137 means the kernel killed the process for exceeding its memory limit.

When the app crashes on its own, its log almost always names the cause. kubectl logs --previous shows the last lines before exit, usually fatal, panic, or refusing to start.

Weekends give a clue here too. Night maintenance rarely ends in a clean shutdown: a node might get rebooted hard, the process gets SIGKILL and never gets to clean up after itself. Whatever it cleans up on a SIGTERM exit stays behind on the persistent volume, and the next start trips over it.

You can't kubectl exec into a container that crashes within a second. To fix data on the volume, stop the Deployment (kubectl scale deploy <name> --replicas=0), attach the volume to a temporary Pod running busybox and sleep, fix things through kubectl exec into that Pod, and delete it. Then bring the replicas back. With ReadWriteOnce, two Pods on different nodes can't share the volume anyway, and writing to the same data from two processes is risky even on one node, so the service stays down while you fix it.

startupProbe as a separate tool

startupProbe has been stable since Kubernetes 1.20. Until it passes, liveness and readiness don't run at all. The app warms up in peace. After its first success, startupProbe stops running and the regular liveness check takes over. Good fit for a long start: Java, ML models, index warmup.

startupProbe:
  httpGet: { path: /healthz, port: 80 }
  periodSeconds: 5
  failureThreshold: 30        # 30 × 5 = 150 seconds to warm up
livenessProbe:
  httpGet: { path: /healthz, port: 80 }
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3         # a normal strict check after warmup

The app gets up to 150 seconds for a cold start, and a container that hangs after warming up still restarts in about 30 seconds.

How to pick the settings

1. Measure the cold start: how many seconds pass from the container starting to the first successful response from the health endpoint. Double it for margin.
2. Measure how long the health endpoint takes right after start, not on a warmed-up app. Set timeoutSeconds with margin above that, usually 3-5.
3. failureThreshold: 3 fits most services.
4. Don't lower periodSeconds: 10 without a reason.
5. Move a start longer than 30 seconds into a startupProbe, and keep liveness strict.

Common mistakes

• Stretching initialDelaySeconds into minutes. The start time grows over time, so the delay keeps needing to go up again, and a hang in the first few minutes isn't caught at all.
• Fixing the port number without reading the Unhealthy message. context deadline exceeded is about a slow response, not a closed port.
• Moving the check to / because it responds fast. The restarts stop, but liveness stops seeing the hang.
• Removing livenessProbe entirely. A hung process then lives on until someone intervenes by hand.
• Editing the check on a running Pod. A Pod's probe fields can't be changed, edit the Deployment template instead.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-weekend-crashloop     # set up the starting state
./run.sh verify k8s-weekend-crashloop    # check the current step

In short

• Crashes when there's no load point to a livenessProbe that can't survive a cold start.
• Exit Code: 137 plus a Killing ... failed liveness probe event means kubelet restarted the container; the app didn't crash on its own.
• The tail of the Unhealthy message tells a closed port from a slow response.
• kubectl logs --previous shows what happened before the restart.
• No Killing event and Exit Code: 1: the app exited on its own, look for the cause in its log, not in the probe.
• The time to restart is no more than initialDelaySeconds + periodSeconds × failureThreshold.
• A long start is fixed with startupProbe, a slow response with the right timeoutSeconds, not with an endless initialDelaySeconds.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl get pods: the STATUS and RESTARTS columns.
• kubectl describe pod: the Liveness: line (or its absence), the Last State block with Reason and Exit Code, and the events.
• kubectl logs <pod> --previous.
• kubectl scale deploy <name> --replicas=N.
• A temporary Pod with a PVC: kubectl apply -f - with a Pod manifest whose volume is a persistentVolumeClaim, then kubectl exec into it and kubectl delete pod.