Job: a one-off task, start to finish

Why

A Deployment assumes "the app runs forever". If the container exits, it gets restarted. A Job works the other way: "run the task to completion and stop." That's for database migrations, backups, a one-off data conversion, generating a report.

What a Job is

A Job is a batch/v1 controller. It creates a Pod from spec.template, waits for exit 0, and marks itself Complete. If the Pod fails with a non-zero code, the Job creates a new Pod and tries again, until it hits backoffLimit.

A Job computes its status from the Pods it created itself. A Job's Pods usually stay in the cluster after finishing, so you can look at their logs. Cleanup happens through ttlSecondsAfterFinished (if set) or by hand. Delete the Job, and its Pods go with it.

Minimal YAML

apiVersion: batch/v1
kind: Job
metadata:
  name: hello-job
spec:
  completions: 1
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: hello
          image: busybox:1.36.1
          command: ["sh", "-c", "echo done; exit 0"]

Fields explained:

• completions: 1. How many successful Pods the Job needs to be considered finished. Set it to 5, and the controller creates five Pods one after another and waits for the fifth success. Handy for sharded tasks.
• parallelism: 1. How many Pods can be running at once. Default is 1. Combines with completions: with completions: 4 and parallelism: 2 the Pods go in two waves of two.
• backoffLimit: 2. How many failed attempts the Job survives before becoming Failed. The default is 6, which is a lot for production. A broken script with the default creates 7 Pods (the first attempt plus six retries).
• restartPolicy. In a Job it's only OnFailure or Never. The apiserver rejects Always. The logic is simple: the Job runs the retries, kubelet doesn't get involved. The field lives in the template, that is, on the Pod, not in the Job's spec.
• activeDeadlineSeconds. A hard timeout for the whole Job, not a Pod. Once it passes, the Job kills all its Pods and becomes Failed.
• ttlSecondsAfterFinished. Delete the Job (and its Pods) N seconds after it finishes, successfully or not. Without it, the garbage piles up.

You can look up the description of any field straight from the cluster:

kubectl explain job.spec
kubectl explain job.spec.backoffLimit
kubectl explain job.spec.template.spec.restartPolicy

restartPolicy: Never vs OnFailure

With OnFailure, kubelet restarts the container itself on the same Pod. With Never, the Job creates a new Pod for every retry. The behavior looks different: with OnFailure, kubectl get pods shows one Pod with several restarts; with Never, several Pods. Never is easier to debug with: every attempt has its own Pod and its own logs.

restartPolicy	Who restarts	What you see in kubectl get pods
Never	The Job creates a new Pod	several Pods, one per retry
OnFailure	kubelet restarts the container	one Pod with a growing restart count
Always	nobody, the manifest is rejected	apiserver rejects the Job

Trying to create a Job with restartPolicy: Always ends in a validation error:

The Job "always" is invalid: spec.template.spec.restartPolicy: Required value: valid values: "OnFailure", "Never"

How a Job counts attempts

The backoffLimit counter isn't reset by a success and doesn't decay over time. It counts failed attempts across the whole Job: failed Pods with Never, and container restarts with OnFailure. Between attempts the Job waits longer each time: 10 seconds, 20, 40, and so on, up to 6 minutes.

Here's a Job with restartPolicy: Never and backoffLimit: 2, whose script exits with code 1 right away:

NAME              READY   STATUS   RESTARTS   AGE
failnever-4kq7d   0/1     Error    0          75s
failnever-9xw2m   0/1     Error    0          64s
failnever-tz8rb   0/1     Error    0          43s

Three Pods: the first attempt and two retries. After the third failure the Job gives up:

NAME        STATUS   COMPLETIONS   DURATION   AGE
failnever   Failed   0/1           75s        75s

The reason is visible in the Job's conditions (kubectl describe job <name>, the Conditions block) and in events: BackoffLimitExceeded means the attempts ran out, DeadlineExceeded means activeDeadlineSeconds kicked in.

Job failed: what's next

Logs first. The Pods of the failed attempts stick around, so you can read the output of every attempt:

kubectl logs -l job-name=hello-job --prefix

The Job itself sets the job-name label (and batch.kubernetes.io/job-name) on its Pods. The --prefix flag tags each line with its Pod's name.

Then comes the trap. A Job in state Failed doesn't start anything else, even after you've fixed the cause: patched the ConfigMap, created the Secret, brought the database up. There won't be new Pods.

Editing the Job to "restart" it doesn't work either: a Job's spec.template is immutable.

The Job "hello-job" is invalid: spec.template: Invalid value: ...: field is immutable

So a Job is restarted by recreating it: delete it and create it again from the same manifest.

kubectl delete job hello-job
kubectl apply -f hello-job.yaml

Same thing in one command: kubectl replace --force -f hello-job.yaml. Keep the manifest in a file. If you dump a live Job with kubectl get job -o yaml and apply it back, the apiserver will refuse: the dump still has the old Job's selector and labels with controller-uid.

What kubectl get jobs shows

kubectl create job hello --image=busybox:1.36.1 -- sh -c 'echo done'
kubectl get jobs

NAME    STATUS     COMPLETIONS   DURATION   AGE
hello   Complete   1/1           5s         10s

STATUS can be Running, Complete, Failed. COMPLETIONS shows successful Pods out of the needed count. Other useful commands:

kubectl get pods -l job-name=hello
kubectl logs -l job-name=hello
kubectl wait --for=condition=complete job/hello --timeout=60s
kubectl delete job hello

Common mistakes

• restartPolicy: Always on a Job, or restartPolicy at the Job's spec level instead of the template. The apiserver rejects it, and the error message is worth reading closely.
• backoffLimit set too high. A buggy script hammers the database with retries.
• Expecting a Failed Job to come back to life once the config is fixed. It won't, it has to be recreated.
• kubectl edit job to tweak the command or image. The template is immutable, it needs recreating.
• Forgetting ttlSecondsAfterFinished. A week later the namespace has thousands of finished Jobs, and kubectl slows down.
• A long database migration in an init container instead of a Job. Every Pod restart runs it again. A migration should be a Job that rolls out once.

What you'll need for the tasks

• Job fields: completions, parallelism, backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished, and restartPolicy in the template.
• kubectl get jobs, kubectl describe job <name> (the Conditions block).
• kubectl get pods -l job-name=<name> and kubectl logs -l job-name=<name> --prefix.
• Recreating a Job: kubectl delete job and kubectl apply -f.
• kubectl get configmap <name> -o yaml. Through envFrom, every ConfigMap key becomes an environment variable with the same name.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your computer, instructions in its README.md.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-first-job     # set up the starting state
./run.sh verify k8s-first-job    # check the current step

In short

• A Job runs a Pod until it succeeds and then stops, unlike a Deployment.
• completions sets the number of successes needed, parallelism how many run at once.
• backoffLimit caps the failed attempts for the whole Job; the default of 6 is too high for production.
• restartPolicy is only Never or OnFailure, Always is forbidden.
• A Failed Job doesn't restart itself, and the template is immutable: recreate it.
• ttlSecondsAfterFinished gets rid of the graveyard of finished Jobs.