kubectl logs and the multi-container Pod

Why this matters

A Pod crashed, restarted and is running again, but monitoring only shows a restart counter. You can almost always see why it crashed in the logs: the application had time to write an error before exiting. You need to know where kubectl logs gets them from, which container it shows by default, and how to pull the output of the previous run.

How Kubernetes collects logs

A container writes to stdout and stderr. The container runtime (containerd or CRI-O) intercepts both streams and writes them to a file on the node: /var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log. Kubelet reads this file. kubectl logs sends a request to the API server (GET /api/v1/namespaces/<ns>/pods/<pod>/log), the API server forwards it to kubelet on the right node, and kubelet returns the file's contents.

Two things follow from this.

First: if the application writes its log to a file inside the container, kubectl logs won't see it. The log has to go to stdout or stderr. That's how the nginx image works, for example: in it, /var/log/nginx/access.log is a symlink to /dev/stdout.

Second: the file on the node gets rotated. By default kubelet starts a new file after 10 MiB (containerLogMaxSize) and keeps up to 5 files per container (containerLogMaxFiles). kubectl logs only returns the latest file, so at most 10 MiB. For long-term storage, people run an external collector (Loki, Elasticsearch, Fluent Bit), which is a topic for separate lessons.

kubectl logs: the main flags

Basic command:

kubectl logs <pod>

Instead of a Pod name you can give a Deployment: kubectl logs deploy/web. kubectl will pick one of its Pods.

Useful flags:

• -c <container> picks a container in a Pod that has more than one.
• -f (follow) streams new lines until you press Ctrl+C.
• --tail=N shows the last N lines. Without a selector kubectl returns the whole file, which is a lot for a long-lived Pod.
• --since=10m shows entries from the last 10 minutes. An exact time is set with --since-time=2026-01-01T00:00:00Z.
• --previous (-p) shows the log of the container's previous run, details below.
• --timestamps adds the record time to every line.
• -l app=web collects logs from all Pods matching a selector. With a selector, --tail defaults to 10, not all lines. With -f, kubectl follows at most 5 Pods at once (the --max-log-requests flag).

kubectl logs <pod> --tail=50 --timestamps
kubectl logs <pod> -c <container> --previous
kubectl logs -l app=web --tail=20 --prefix

The multi-container Pod

A Pod can hold several containers. They share the network and volumes, but each has its own stdout, and its own log in a separate file.

Pod my-pod
 ├─ app      own stdout   -c app
 └─ sidecar  own stdout   -c sidecar

Standard patterns:

• Sidecar. A helper for the main container. For example, next to nginx a log-shipper that reads the access log from a shared volume and sends it to Loki. Since Kubernetes 1.33 (GA), this kind of sidecar is usually described as an initContainer with restartPolicy: Always: it starts before the main container and stops after it. For kubectl logs this makes no difference, -c <name> works the same way.
• Adapter. Translates the main container's output into the format you need.
• Ambassador. A proxy the main container uses to reach external services.

If you don't give -c, kubectl doesn't error out. It picks a default container: the one named in the Pod annotation kubectl.kubernetes.io/default-container, or without it, the first one in the containers list. It prints one line about this to stderr:

Defaulted container "report" out of: report, log-shipper

This line is easy to miss. Then you're reading the sidecar's log and not finding the application's error in it. That's why in a multi-container Pod it's better to name the container explicitly:

kubectl logs my-pod -c app
kubectl logs my-pod -c sidecar

Everything from every container in the Pod at once, including init containers, comes from --all-containers:

kubectl logs my-pod --all-containers --prefix

--prefix adds a source tag to every line, like [pod/my-pod/app]. Without it, lines from different containers get mixed together.

--previous and why you need it

When a container crashes, kubelet starts it again in the same Pod and writes the new run's log to a new file (0.log, then 1.log, and so on). A plain kubectl logs shows the current run. The error the container crashed with is left in the previous file, and --previous pulls it out:

kubectl logs my-pod --previous
kubectl logs my-pod -c app --previous

Limits:

• kubelet keeps the log of only one completed run. The one before that is gone.
• If the Pod was deleted or recreated (a new Pod from a Deployment, eviction from a node), the new Pod has no previous runs. The old Pod's logs went away with it.
• If the container has never restarted, --previous returns an error: there is no previous run.

How many times a container has restarted shows up in the RESTARTS column of kubectl get pods, and per container in kubectl describe pod (the Restart Count field and the Last State block with the exit code).

Common mistakes

• Reading the log without -c in a Pod with a sidecar, and missing the Defaulted container line. There's no application error in the output, because it's another container's log.
• Looking at the current log of a crashed container. The new run hasn't reached the error yet, and the cause is in --previous.
• Waiting in kubectl logs for a log the application writes to a file inside the container. It won't show up there until the output is sent to stdout or stderr.
• Using -l and being surprised the lines are few. With a selector, --tail defaults to 10.
• Relying on kubectl logs as storage. History is limited to the last file on the node and disappears along with the Pod.

What you'll need for the tasks

• kubectl get pods and the RESTARTS column: which Pod is restarting.
• kubectl describe pod: which container is restarting and with what exit code.
• kubectl logs <pod> -c <container> and --previous.
• kubectl logs --all-containers --prefix.
• Understanding that kubectl logs only sees stdout and stderr.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-logging     # set up the starting state
./run.sh verify k8s-logging    # check the current step

In short

• Logs are a container's stdout and stderr, which the runtime writes to a file on the node. kubectl logs reads them through the API server and kubelet.
• Anything written to a file inside the container never reaches kubectl logs.
• Rotation defaults to 10 MiB × 5 files, kubectl logs only sees the latest file.
• In a multi-container Pod without -c, kubectl picks a default container and warns with a Defaulted container line.
• --previous shows one previous run of the container in the same Pod.