Sidecar for logs

Why this matters

An application writes logs to a file, and something has to ship them out: to Loki, Elasticsearch, or just to stdout, where the cluster collects them. Building that shipping into the application itself is inconvenient: every team has its own language and its own agent. It's easier to put a separate container next to it that reads the same file and only handles delivery.

Such a container is called a sidecar. Besides logs, mTLS proxies, config loaders, and monitoring agents are built the same way.

What a sidecar is

A sidecar is an extra container in the same Pod that helps the main one. Not a separate Pod behind a Service, but a neighboring container inside the Pod.

Containers of one Pod share the network and can share volumes. The sidecar reads the same file the main container writes to, and sees the same 127.0.0.1. No Service, DNS, or network hops are needed for this.

A classic example: nginx writes the access log to a file, and the sidecar log-shipper reads it with tail -F and prints it to its own stdout. On busybox this is a one-line command. In production, fluent-bit, filebeat, or vector replace busybox.

Where shared data lives

A volume gives containers of a Pod a shared filesystem. Most often that's emptyDir: an empty directory created when the Pod starts and deleted with it. Both containers mount this volume, each at its own mountPath, and see the same files.

spec:
  volumes:
    - name: logs
      emptyDir: {}
  containers:
    - name: nginx
      image: nginx:1.27.3-alpine
      volumeMounts:
        - name: logs
          mountPath: /var/log/nginx
    - name: log-shipper
      image: busybox:1.36.1
      command: ["sh", "-c", "tail -F /var/log/nginx/access.log"]
      volumeMounts:
        - name: logs
          mountPath: /var/log/nginx

Containers can use different paths. What matters is the volume name: the file nginx writes to /var/log/nginx/access.log shows up for the sidecar at whatever path it mounts the same logs volume to.

emptyDir lives exactly as long as the Pod. The Pod gets recreated, the data is gone. For logs that's fine: the sidecar ships them out while the Pod is alive.

Sidecar and startup order

Containers from the containers list start with no guaranteed order. kubelet doesn't guarantee that nginx will create the file before the sidecar tries to open it. That's why sidecars are written with tail -F (capital F), not tail -f.

With tail -f, busybox prints tail: no files and exits with code 1 when the file is missing. The sidecar crashes, kubelet restarts it, the Pod ends up in CrashLoopBackOff, even though nginx is working fine. tail -F waits for the file to appear, then reads it:

tail: can't open '/var/log/nginx/access.log': No such file or directory
tail: /var/log/nginx/access.log has appeared; following end of new file

Native sidecar

A native sidecar is an init container with restartPolicy: Always. It starts before the main containers, runs for the whole life of the Pod, and stops after them. Support is on by default since Kubernetes 1.29, and became stable (GA) in 1.33. On a current cluster this is the main way to describe a sidecar.

spec:
  initContainers:
    - name: log-shipper
      image: busybox:1.36.1
      restartPolicy: Always
      command: ["sh", "-c", "tail -F /var/log/nginx/access.log"]
      volumeMounts:
        - name: logs
          mountPath: /var/log/nginx

For an ordinary init container, kubelet waits for it to exit with code 0. For a native sidecar, it only waits for it to start successfully and immediately moves on to the next one. A native sidecar counts toward the READY column: a Pod made of nginx and such a sidecar shows 2/2 there.

Writing a sidecar in the plain containers list still works, old manifests are full of it. But the startup order there isn't defined, and when the main container finishes, that kind of sidecar doesn't find out and keeps running. For a Deployment this goes unnoticed. For a Pod that's supposed to finish (a Job), it's a problem: as long as even one container from containers is running, the Pod isn't done.

	in containers (legacy)	in initContainers with restartPolicy: Always
Startup order	not defined, parallel with the main containers	starts before the main containers
When the main container exits	keeps running	kubelet stops it after the main containers
In the READY column	counted	counted
Status of the form	common in old manifests	the main approach, GA since 1.33

Related commands

kubectl logs <pod> -c <container>       # log of one specific container
kubectl logs <pod> --all-containers     # logs of all containers in a row
kubectl exec <pod> -c <container> -- sh
kubectl describe pod <pod>              # both containers and their statuses
kubectl get pod <pod> -o jsonpath='{.spec.initContainers[*].name}'

kubectl logs <pod> without -c on a Pod with several containers takes the first one from containers and warns about it:

Defaulted container "nginx" out of: nginx, log-shipper (init)

It's easy to miss the sidecar's log this way: check which container kubectl picked.

Why you can't add a container to an existing Pod

Most fields of a Pod spec can't be changed after creation. The containers and initContainers lists are fixed, though a running Pod's image, for example, can be swapped. Trying to add a container ends with the API refusing it:

The Pod "report-5zljr" is invalid: spec.containers: Forbidden: pod updates may not add or remove containers

To add a sidecar, you change the template in the Deployment. The controller creates new Pods with the updated spec and deletes the old ones. A Job's Pod template is also immutable: you delete the Job and create it again.

What commonly breaks

• The sidecar mounted the volume, but the main container didn't (or the other way around). There's no error, the sidecar just waits for a file that never shows up.
• tail -f instead of tail -F: the sidecar crashes on start if the file doesn't exist yet.
• Forgot -c in kubectl logs: kubectl showed the log of a different container.
• nginx from the official image starts as root, and runAsNonRoot: true keeps it from starting. nginxinc/nginx-unprivileged works instead, it listens on 8080 and runs as uid 101.
• Sidecar in containers on a Job: the main container finished, but the Job doesn't complete because the sidecar keeps running.
• A busybox sidecar running sh -c "tail -F ..." acts as PID 1 and doesn't respond to SIGTERM. kubelet waits out terminationGracePeriodSeconds (30 seconds by default) before killing it. The Pod and Job complete with that delay. A signal handler removes the pause: trap 'exit 0' TERM; tail -F /var/log/app.log & wait.

What you'll need for the tasks

• volumes with emptyDir and volumeMounts on every container that shares files.
• Native sidecar: initContainers with restartPolicy: Always and tail -F inside sh -c.
• kubectl logs <pod> -c <container> and kubectl describe pod.
• Editing a Deployment's template (kubectl apply -f or kubectl edit deploy).
• Job status in kubectl get job and recreating a Job: kubectl delete job and kubectl apply -f.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-sidecar-logger     # set up the starting state
./run.sh verify k8s-sidecar-logger    # check the current step

In short

• A sidecar is an extra container in the same Pod, sharing its network and volumes.
• Containers exchange files through emptyDir, which lives exactly as long as the Pod.
• tail -F survives the file being absent at start, tail -f doesn't.
• A native sidecar is written as an initContainer with restartPolicy: Always, GA since 1.33. It starts first and stops last.
• The set of containers in an existing Pod can't be changed, you need a new Pod: a Deployment rollout or recreating a Job.