Init containers

Why this matters

An app often needs to do something before it starts: wait for a database, download files, build a config from a template. You could hide that in a bash wrapper inside the main image. Then the app's image carries utilities and logic it doesn't need once it's running.

Kubernetes gives preparation its own place: init containers. They run before the main containers, each with its own image and its own command. If the preparation fails, the app never starts at all, and you can see why in the Pod's status.

What an initContainer is

An init container (initContainer) is a Pod container that has to exit with code 0 before the main containers start. Only after that does kubelet bring up the regular containers.

The rules:

• init containers run strictly in order, in the order they're listed;
• the next one starts only once the previous one exited with code 0;
• if an init container fails, the main containers don't start, and kubelet retries the init container with a growing backoff (when the Pod's restartPolicy is Always or OnFailure, which for a Deployment is always Always).

An init container and a main container can share an emptyDir volume: the init container puts files there, the main container reads them. That's how you pass downloaded assets and generated configs between them.

Minimal YAML

A Deployment whose main container waits for Service db on port 5432:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      initContainers:
        - name: wait-for-db
          image: busybox:1.36.1
          command:
            - sh
            - -c
            - 'until nc -z -w 2 db 5432; do echo waiting; sleep 2; done'
      containers:
        - name: main
          image: busybox:1.36.1
          command: ["sh", "-c", "sleep 3600"]

initContainers sits in spec.template.spec next to containers and is described with the same fields: name, image, command, volumeMounts.

nc -z host port tries to open a TCP connection and closes it right away, without sending any data. If the connection doesn't open, nc exits with a nonzero code, and until tries again. It's the cheapest way to "wait for a port", and it works in any busybox. The -w 2 flag caps one attempt at two seconds. Without it, an attempt against an unreachable host can hang for a long time.

Init doesn't count toward READY

The STATUS column in kubectl get pod shows which init container the Pod is on right now. Here's a Pod with two init containers, the second one waiting on a database:

NAME     READY   STATUS            RESTARTS   AGE
phases   0/1     Init:0/2          0          5s
phases   0/1     Init:1/2          0          15s
phases   0/1     PodInitializing   0          35s
phases   1/1     Running           0          42s

Init:1/2 means one of the two init containers has already finished. PodInitializing shows up once all init containers have passed and the main containers are still starting.

A regular init container doesn't go into the READY count: the Pod above has two init containers and one main container, and the column still reads 1/1. readinessProbe isn't supported on a regular init container: the API rejects such a manifest with the error may not be set for init containers without restartPolicy=Always. The exception is the sidecar form (see below): its probes do work, and it counts toward READY.

If an init container fails, the status becomes Init:Error, and after a few retries, Init:CrashLoopBackOff.

Check an init container's status separately from the main containers:

kubectl get pod <name> -o jsonpath='{.status.initContainerStatuses[*].name}'
kubectl describe pod <name>

In describe, init containers have their own Init Containers section with State, Last State, and Exit Code fields.

What usually goes in init containers

• Waiting for a service, as in the example above, with nc.
• Downloading static assets into an emptyDir, served by the main container.
• Generating a config: the init container takes a template and environment variables, runs them through envsubst, and writes the result to an emptyDir.
• Preparing a volume: the init container sets permissions on a shared volume, and the main container starts once it's ready.
• Copying certificates from a Secret or an external store to the right place.

restartPolicy: Always turns init into a sidecar

An init container has its own restartPolicy field. Set it to Always, and the container stops being a one-shot. It starts before the main containers, runs for the Pod's whole lifetime, and stops after them. This is a native sidecar, a stable feature since Kubernetes 1.33.

initContainers:
  - name: log-shipper
    image: busybox:1.36.1
    restartPolicy: Always
    command: ["sh", "-c", "tail -F /var/log/app/app.log"]

How a sidecar differs from a regular init container:

• the Pod doesn't wait for it to finish, a successful start is enough (or, if a startupProbe is set, passing that);
• its liveness, readiness, and startup probes work, and it counts toward the Pod's READY;
• if it fails, kubelet restarts it in place, the Pod stays the same.

time ------------------------------>

regular init:
  init-1  [==]
  init-2      [==]
  main            [==============]

init with restartPolicy: Always:
  sidecar [============================]
  main            [==============]

This is how log agents, mesh proxies, and metrics exporters are described. Without restartPolicy, an init container stays regular: it runs, exits, and disappears. More on sidecars in the next lesson.

Init container resources

Init containers run one at a time and before the main ones, so their requests don't add up with the main containers' requests. The scheduler takes the Pod's effective request as the larger of two numbers: the biggest request among the init containers, and the sum of the main containers' requests.

Example: an init container asks for 1500m CPU, the main container for 100m. The Pod will take up 1500m on the node, even though only the main container is actually working after startup. A heavy init container makes the whole Pod heavy.

A sidecar init container behaves like a main container in this count: it runs alongside them, so its request gets added to the sum.

Anti-pattern: database migrations in init

Long database migrations don't belong in an init container. Init runs on every new Pod: on every rollout, every re-creation, every replica, and replicas run it at the same time. While the migration runs, the Pod isn't Ready, and three replicas end up trying to migrate the same database in parallel.

Migrations go in a separate Job that runs once before the Deployment rolls out.

Related commands

kubectl get pod -w                      # shows phases Init:0/1, PodInitializing, Running
kubectl describe pod <name>             # Init Containers section
kubectl logs <pod> -c wait-for-db       # init container's log
kubectl explain pod.spec.initContainers

kubectl logs <pod> without -c shows the main container. While the Pod is still at the init stage, you can't see the init container's log this way:

Defaulted container "main" out of: main, first (init), wait-for-db (init)
Error from server (BadRequest): container "main" in pod "phases" is waiting to start: PodInitializing

What commonly breaks

• restartPolicy: Always on an init container, set without thinking it through. That container becomes a sidecar and never finishes, and the Pod only waits for it to start successfully. Waiting for a dependency stops working. Don't set this field when preparing an environment.
• A large request on an init container. The Pod's effective request is computed as the maximum, and the Pod may not fit on a node even though the main container is small.
• No nc or shell in the image. Waiting for a port needs busybox or another image with utilities, distroless won't work.
• A wait loop without sleep: it works, but it burns CPU and floods the log.

What you'll need for the tasks

• Where initContainers sits in a Deployment manifest and what fields it has.
• The loop until nc -z <host> <port>; do ...; sleep 2; done inside sh -c.
• Reading the statuses Init:0/1, Init:1/2, Init:Error, Init:CrashLoopBackOff.
• kubectl logs <pod> -c <init> and the Init Containers section in kubectl describe pod.
• A shared emptyDir volume between an init container and the main container: volumes on the Pod and volumeMounts on each container.
• kubectl scale and kubectl edit deploy.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-init-container     # set up the starting state
./run.sh verify k8s-init-container    # check the current step

In short

• An init container runs before the main containers and has to exit with code 0.
• Init containers run one after another, in the order they're listed.
• Typical jobs: waiting for a dependency, generating a config, preparing a volume.
• A regular init container doesn't count toward READY and doesn't support probes.
• restartPolicy: Always on an init container turns it into a native sidecar with probes and in-place restarts.