livenessProbe restarts the container

The Deployment is rolled out, the image is right, the app inside the container works. And yet the RESTARTS counter keeps growing, and the status flips between Running and CrashLoopBackOff.

NAME                   READY   STATUS             RESTARTS      AGE
app-7c9d8f4b6-x2k4j    0/1     CrashLoopBackOff   4 (12s ago)   2m

A common cause: it isn't a bug in the app killing the container, it's livenessProbe. That's the check kubelet uses to decide whether a container is alive. In this lesson you'll learn to tell this kind of restart apart from a real app crash, and find where the check is pointing.

What kubectl shows

There are three places worth checking, in this order.

1. kubectl get pod

The first thing you notice: RESTARTS is above zero and growing. In parentheses, kubectl shows how long it's been since the last restart.

READY 0/1 means the container isn't ready to take traffic right now: it either failed readinessProbe (the readiness check) or isn't running at all and is waiting for the next attempt. Four restarts in two minutes is already a pattern, not a one-off error.

2. kubectl describe pod

This command shows three things get doesn't:

• Events at the end: what happened to the Pod. The API server keeps events for a limited time, an hour by default (the --event-ttl flag).
• State and Last State: what's happening with the container now, and why it exited last time.
• Containers: the settings that actually took effect (probe, ports, environment variables).

Typical output when liveness is broken:

    State:          Running
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
    Liveness:       http-get http://:8081/ delay=0s timeout=1s period=10s #success=1 #failure=3
...
Events:
  Type     Reason     Age   From     Message
  Warning  Unhealthy  5s    kubelet  Liveness probe failed: Get "http://10.244.0.7:8081/": dial tcp 10.244.0.7:8081: connect: connection refused
  Normal   Killing    5s    kubelet  Container app failed liveness probe, will be restarted

Exit code 137 is 128 + 9: the process was terminated by the SIGKILL signal. After failureThreshold failed checks in a row (3 by default), kubelet stops the container: it sends SIGTERM first, and if the process hasn't exited within terminationGracePeriodSeconds, it finishes it off with SIGKILL. Many processes that run as PID 1 in a container don't handle SIGTERM, which is why the exit code is usually 137.

The key part of Events: connection refused at an address with port 8081. The app isn't listening on that port.

3. kubectl logs --previous

kubectl logs <pod> shows the container's current run. If it just restarted, the output will have fresh lines, not the ones from when the problem happened. The --previous flag (or -p) gives you the log of the previous run.

kubectl logs app-7c9d8f4b6-x2k4j --previous

A line like listening on 0.0.0.0:8080 next to a probe hitting :8081 settles the question of the cause.

States that cause confusion

Status	What it means
Pending	The Pod isn't scheduled to a node yet, or the image is still downloading. Check Events.
Running	The container is running. Whether it's ready for traffic is a separate question, shown by READY.
CrashLoopBackOff	The container keeps exiting over and over, and kubelet increases the pause between restarts (10s, 20s, 40s, and so on, up to 5 minutes).
Error	The container exited with a nonzero code. With restartPolicy: Never the status stays; for a Deployment it flickers between restarts.
Completed	The container exited with code 0. Normal for a Job, bad for a Deployment: the container gets restarted and ends up in CrashLoopBackOff.
ImagePullBackOff	The image won't download: a typo in the name, no access, no network.
CreateContainerConfigError	The Pod references a ConfigMap or Secret that doesn't exist.
OOMKilled	The container went over resources.limits.memory, and the kernel killed it. Shows up in Last State.

Probe vs. the app's real state

livenessProbe answers the question "is the container alive?" If the check fails failureThreshold times in a row, kubelet restarts the container. From there, two scenarios are possible.

First: the app really did hang. The probe did its job, and it's the app that needs fixing.

Second: the probe is misconfigured. The app is alive, but the check is pointing the wrong way: wrong port, wrong path, too short a pause for a slow startup. This case is more common. For example, the container listens on 8080 (that's how unprivileged nginx images work), but the probe still has port 80 left over from the template.

A wrong path shows up differently in Events than a wrong port. A closed port: connect: connection refused. An open port with no such path: HTTP probe failed with statuscode: 404. For httpGet, codes from 200 to 399 count as success.

How to read Events

kubectl describe pod ends with a table:

Events:
  Type     Reason     Age                From     Message
  Normal   Started    40s (x2 over 52s)  kubelet  Started container app
  Warning  Unhealthy  31s (x6 over 49s)  kubelet  Liveness probe failed: ...
  Normal   Killing    31s (x2 over 43s)  kubelet  Container app failed liveness probe, will be restarted
  Normal   Pulled     30s (x3 over 52s)  kubelet  Container image "busybox:1.36.1" already present on machine

kubectl merges identical events into one line: x6 over 49s means the event happened six times over 49 seconds. So the line order doesn't match the actual timeline. Go by the counters instead: how many times the container started (Started), how many times it was killed (Killing), and why. The Unhealthy reason with the words Liveness probe failed points straight at the probe.

If the container exits on its own, the picture is different. Last State shows OOMKilled (ran out of memory) or Error with the app's exit code, and there are no liveness-related Killing events. Instead you get BackOff: kubelet waiting before the next start attempt.

How it's fixed

Check the port in the probe against the port the app is actually listening on:

kubectl get deploy app -o jsonpath='{.spec.template.spec.containers[0].livenessProbe}'

{"failureThreshold":3,"httpGet":{"path":"/","port":8081,"scheme":"HTTP"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1}

Then edit the Deployment template, not the Pod. You can't change the probe on a running Pod, and a Pod created by a Deployment gets replaced with a new one anyway.

kubectl patch deploy app --type=json \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/port","value":8080}]'

Or use kubectl edit deploy app. The Deployment creates a new ReplicaSet, the old Pods go away, and the new ones keep RESTARTS at zero. Wait a few check cycles (periodSeconds × failureThreshold) and look again: zero after that is the sign the fix worked.

In production the Deployment usually comes from Helm, Kustomize, or Argo CD. Then kubectl patch is only an emergency fix, and the permanent one goes into the manifest.

Common mistakes

Reading logs without --previous. The current run just started; there's nothing useful in it. You need the log of the run that got killed.

Removing livenessProbe to make the restarts stop. They'll stop, but now nothing notices a real hang either. You fix the check, you don't turn it off.

Raising failureThreshold or initialDelaySeconds to huge numbers. If the probe is hitting a closed port, it will never pass. All you're doing is delaying the restart.

Confusing liveness with OOM. Exit code 137 happens in both cases. What tells them apart is the Reason in Last State (OOMKilled or Error) and the Killing events.

Editing the Pod instead of the Deployment. kubectl delete pod just gets you a new Pod with the same broken probe.

How to avoid it

The probe's port and containerPort should come from the same place. When the manifest uses a named port, the probe references the name, not the number:

ports:
  - name: http
    containerPort: 8080
livenessProbe:
  httpGet:
    path: /healthz
    port: http

That way, changing the port in one place won't leave the probe with a stale value.

Second rule: liveness and readiness check different things. Readiness pulls the Pod out of load balancing while it isn't ready, and doesn't kill the container. Liveness does.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start debug-probe     # set up the starting state
./run.sh verify debug-probe    # check the current step

In short

• A growing RESTARTS on a live app almost always points to livenessProbe.
• Exit code 137 means SIGKILL. Whether it came from kubelet after failed checks or from the kernel over OOM shows up in Last State and Events.
• The error text is in Events: Liveness probe failed plus the address and port, or the response code.
• kubectl logs --previous shows the log of the killed run; plain logs no longer has it.
• You fix the Deployment template, after which new Pods start with zero restarts.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl get pods: the READY, STATUS, RESTARTS columns.
• kubectl describe pod <pod>: Last State (Reason and Exit Code), the Liveness: line, and the Events block.
• kubectl logs <pod> --previous.
• Editing the probe in the Deployment template: kubectl edit deploy or kubectl patch deploy.