CronJob: scheduled tasks

Why

Almost every service has recurring tasks: a backup once an hour, cleanup of stale records every night, syncing with an external API every 15 minutes. This used to mean keeping a cron job on a separate VM. In Kubernetes there's an object for this, CronJob: the schedule lives in the cluster next to the application, and the task itself is run by an ordinary Job.

What a CronJob is

A CronJob sits one level above a Job. It doesn't create the Pod itself. On schedule, a CronJob creates a new Job, and the Job controller creates the Pod and waits for it to finish. So everything you already know about Jobs (backoffLimit, restartPolicy, logs from a failed Pod) applies here too.

CronJob is batch/v1, stable since Kubernetes 1.21. The old batch/v1beta1 was removed in 1.25; a new cluster won't accept manifests that use it. The schedule is handled by the CronJob controller inside kube-controller-manager: it computes the next run time and checks each CronJob roughly once every 10 seconds.

Minimal YAML

apiVersion: batch/v1
kind: CronJob
metadata:
  name: hello-cron
spec:
  schedule: "* * * * *"
  successfulJobsHistoryLimit: 2
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: hello
              image: busybox:1.36.1
              command: ["sh", "-c", "echo tick"]

Note the nesting: the CronJob's spec → jobTemplate.spec (the Job's spec) → template.spec (the Pod's spec). Job fields like backoffLimit go in jobTemplate.spec, Pod fields like restartPolicy go one level deeper still.

A CronJob's name can be at most 52 characters. The controller appends a suffix to it to get the Job's name, and a Job's limit is 63 characters.

Field breakdown

• schedule. A regular five-field cron expression: minute, hour, day of month, month, day of week. * * * * * means "every minute", */15 * * * * means "every 15 minutes", 30 3 * * * means "every day at 03:30".
• timeZone. The schedule's time zone, a name from the tz database: Europe/Moscow, Etc/UTC. Stable since 1.27. The API rejects an unknown name (unknown time zone Moscow). You also can't prefix the time zone right in the schedule (CRON_TZ=Europe/Moscow 0 3 * * *): cannot use TZ or CRON_TZ in schedule, use timeZone field instead.
• jobTemplate. The full spec of an ordinary Job. On every run, the controller creates a new Job from it.
• successfulJobsHistoryLimit. How many successful Jobs to keep, default 3. Older ones are deleted along with their Pods.
• failedJobsHistoryLimit. Same, for failed ones, default 1.
• concurrencyPolicy. What to do if the previous Job hasn't finished and it's time to start the next one:
  • Allow (default). Run it in parallel. Risky for backups: two processes hitting the same database.
  • Forbid. Skip this run, let the previous one finish. In the CronJob's events (kubectl describe cronjob) you'll see JobAlreadyActive: Not starting job because prior execution is running and concurrency policy is Forbid.
  • Replace. Delete the running Job and start a new one.
• startingDeadlineSeconds. How many seconds after the scheduled time a run is still allowed to start. If it's later than that (the controller was down, or Forbid was waiting on the previous Job), this run is skipped and the following ones go by schedule as usual. Without this field, there's no deadline. Don't set it below 10 seconds: the controller checks the CronJob roughly once every 10 seconds and might miss the window.
• suspend: true. Temporarily turn off the schedule without deleting the object.

How a CronJob creates a Job

A Job created on schedule is named <CronJob name>-<number>. The number is the scheduled run time in minutes since 1970, so all Jobs from the same CronJob share a prefix:

kubectl get jobs

NAME                    STATUS     COMPLETIONS   DURATION   AGE
slow-allow-29859640     Running    0/1           76s        76s
slow-allow-29859641     Running    0/1           16s        16s
tick-29859640           Complete   1/1           6s         76s
tick-29859641           Complete   1/1           4s         16s

Each Job points back to its CronJob through metadata.ownerReferences. So kubectl delete cronjob also deletes its Jobs, and their Pods along with them.

A few things that aren't obvious:

• concurrencyPolicy only compares a run against active Jobs of its own CronJob. Two CronJobs doing the same task know nothing about each other, and with Forbid on both, they happily run in parallel.
• suspend: true doesn't touch a Job that's already running: it runs to completion. Only the creation of new ones is turned off.
• Editing a CronJob affects future Jobs. Jobs already created, and their Pods, keep running with the old template.
• If the controller missed several runs (downtime, suspend), it doesn't catch up on each one separately: it creates a single Job for the most recent missed moment. If there are more than 100 missed runs and startingDeadlineSeconds isn't set, the controller doesn't start anything and logs a too many missed start times error. With a deadline set, it only counts misses within that window.
• The schedule is approximate. In rare cases the controller can create two Jobs for one moment, or none at all, so your task needs to tolerate being run twice.

restartPolicy inside jobTemplate

Only OnFailure or Never, same as an ordinary Job. The API rejects Always:

spec.jobTemplate.spec.template.spec.restartPolicy: Required value: valid values: "OnFailure", "Never"

The logic is the same as for a Job: the Pod has to finish at some point, and the Job controller handles retries.

Related commands

kubectl create cronjob hello --image=busybox:1.36.1 --schedule='*/5 * * * *' -- echo hi
kubectl get cronjob
kubectl get jobs --watch
kubectl create job --from=cronjob/hello hello-manual  # run it manually, without waiting for the schedule
kubectl patch cronjob hello -p '{"spec":{"suspend":true}}'
kubectl describe cronjob hello

kubectl create cronjob without flags sets restartPolicy: OnFailure and the defaults: Allow, history limits 3 and 1. Output of kubectl get cronjob:

NAME        SCHEDULE     TIMEZONE        SUSPEND   ACTIVE   LAST SCHEDULE   AGE
tz-moscow   30 3 * * *   Europe/Moscow   False     0        <none>          0s

ACTIVE shows how many Jobs from this CronJob are running right now, LAST SCHEDULE shows how long it's been since the last scheduled run.

A manual run via create job --from creates a Job right away, from the same jobTemplate. Such a Job carries the annotation cronjob.kubernetes.io/instantiate: manual and the same owner CronJob. This is the main way to test a nightly CronJob during the day. When you do this, the CronJob's events will show Warning UnexpectedJob ... Saw a job that the controller did not create or forgot: the controller is reporting that it didn't create this Job. That's expected for a manual run.

Common mistakes

• Forgetting timeZone and assuming the schedule runs on your own clock. A backup meant for "3 AM" ends up running at 6 AM Moscow time, right in the middle of peak load.
• concurrencyPolicy: Allow on a heavy task. The previous run is still going, a new one already started, and two processes hammer the same database and block each other.
• failedJobsHistoryLimit: 0 for tidiness. A failed Job is deleted along with its Pod, and there's nowhere left to see the error logs. The opposite extreme: large limits "for history" pile up hundreds of finished Jobs in the namespace.
• Leaving a * * * * * schedule in production after testing. A backup every minute floods your notifications in a day.
• Expecting suspend to stop a run in progress. It won't: a running Job has to be deleted separately.

What you'll need for the tasks

• CronJob fields: schedule, timeZone, concurrencyPolicy, startingDeadlineSeconds, successfulJobsHistoryLimit, failedJobsHistoryLimit, suspend, and restartPolicy in the Pod template.
• kubectl get cronjob, kubectl get jobs, kubectl get pods -l <label>.
• A Job's owner: kubectl get job <name> -o yaml, field metadata.ownerReferences.
• Manual run: kubectl create job --from=cronjob/<name> <Job name>.
• kubectl explain cronjob.spec.<field> to look up any field.

The labs are in the labs/ folder: each one brings up its own kind cluster on your machine, with instructions in its README.md.

Homework

Practice the theory on your own cluster: 3 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-cronjob     # set up the starting state
./run.sh verify k8s-cronjob    # check the current step

In short

• A CronJob doesn't create a Pod: on schedule it creates a Job, and the Job creates the Pod.
• Without spec.timeZone, the schedule is evaluated using the kube-controller-manager's clock, usually UTC. The field is stable since 1.27.
• The default history limits are 3 and 1; you can raise or lower them.
• concurrencyPolicy (Allow, Forbid, Replace) applies within a single CronJob.
• startingDeadlineSeconds sets how late a run is allowed to start.
• suspend: true turns off the schedule without touching Jobs already running.
• kubectl create job --from=cronjob/... runs a CronJob manually.