Command and args in the container spec

Why this matters

The same image gets started differently: nginx with a different directive, busybox with a debug loop, a utility with different flags. You don't need to rebuild the image for that. Kubernetes lets you override, in the manifest, the program the container runs and its arguments.

A mistake in these fields shows up the same way every time: the Pod starts and immediately crashes or exits. To find the cause quickly, you need to understand how the manifest's fields line up with the image's settings.

What command and args are

Every Docker image has two startup settings. ENTRYPOINT is the main executable. CMD is the arguments to it. Without an override, the container runs ENTRYPOINT + CMD.

In Kubernetes these two settings are overridden by the command and args fields in the container spec. The mapping is one-to-one: ENTRYPOINT in the Dockerfile corresponds to the command field, CMD corresponds to the args field.

A set command overrides ENTRYPOINT. A set args overrides CMD. You can set just one field, both, or neither. If neither is set, whatever is baked into the image runs.

docker image inspect shows what's baked into the image:

docker image inspect redis:7.4.1-alpine --format 'EP={{json .Config.Entrypoint}} CMD={{json .Config.Cmd}}'

EP=["docker-entrypoint.sh"] CMD=["redis-server"]

Minimal YAML

containers:
  - name: shouter
    image: busybox:1.36.1
    command: ["sh"]
    args: ["-c", "echo hi; sleep 3600"]

Inside the container, sh -c 'echo hi; sleep 3600' runs. The elements of command and args get joined into one argument list (argv) of the process. The first element of command becomes argv[0], that is, the program itself, and the rest follow it in order.

Four cases

1. Neither command nor args is set. ENTRYPOINT + CMD from the image runs. The usual path.
2. Only command is set. ENTRYPOINT is overridden, and the image's CMD is dropped entirely. Exactly what's in command runs, with no arguments from the image.
3. Only args is set. ENTRYPOINT comes from the image, the arguments are your own. This is what you do when the image is built correctly and you only need to change the parameters.
4. Both fields are set. The image's startup settings are ignored completely.

The second case is easy to get wrong. It seems like the image's CMD should get appended after your command, but that doesn't happen. An example with busybox, whose CMD is ["sh"]:

command: ["echo"]

The container runs echo with no arguments and prints an empty line. If CMD had been appended, you'd get echo sh.

The third case has a catch too. args replaces CMD entirely, it doesn't get appended to it. If the image's CMD held a program name with flags, and you passed only a new flag in args, the program is no longer in the list. ENTRYPOINT gets the flag as its first argument, and what it does with it depends on the image.

Why busybox exits without arguments

The busybox image has an empty ENTRYPOINT, and CMD is sh. Without arguments the container runs sh. The container's standard input (stdin) isn't connected, sh immediately reads end of input and exits with code 0.

What happens next depends on the Pod's restartPolicy. With Never or OnFailure the Pod stays in Completed status. By default restartPolicy: Always: kubelet restarts the container, kubectl get pod shows Completed with a RESTARTS counter, and after a couple of restarts, CrashLoopBackOff.

NAME        READY   STATUS      RESTARTS      AGE
bb-always   0/1     Completed   2 (31s ago)   53s

The same thing happens with alpine, debian-slim, and any image that runs a shell by default. For the Pod to stay alive, it needs a long-running process: sleep 3600 or an infinite loop like while true; do sleep 30; done.

Quoting and arrays

In YAML, command and args are lists of strings. Each list element becomes a separate argument. So this line:

command: ["sh", "-c", "echo hi; sleep 3600"]

is three arguments: sh, -c, and one string echo hi; sleep 3600. The characters ;, &&, | inside the string are interpreted by sh, not by Kubernetes.

A common mistake:

command: ["sh", "-c", "echo", "hi; sleep 3600"]

This is four arguments. sh -c takes only the next element as the script, which is echo. It runs echo with no arguments and prints an empty line. The rest, hi; sleep 3600, becomes the script's $0 variable and never runs. The container exits in a second.

correct, 3 elements:
["sh","-c","echo hi; sleep 3600"]
  sh -c gets the whole script

wrong, 4 elements:
["sh","-c","echo","hi; sleep 3600"]
  sh -c only gets  echo
  the rest ends up in $0

If in doubt, put the whole script together as one string after sh -c and don't split it up. That leaves less room to break the escaping.

Variables in command and args

In command and args Kubernetes substitutes the container's environment variables in the form $(VAR). It leaves ${VAR} alone, the shell expands that if the command runs through sh -c. $$(VAR) gives the literal text $(VAR).

env:
  - name: GREETING
    value: privet
command: ["echo"]
args: ["$(GREETING)", "${GREETING}", "$$(GREETING)"]

privet ${GREETING} $(GREETING)

Here echo runs without a shell, so ${GREETING} reached the output as is.

command and args don't change on a created Pod

The command and args fields can't be changed on a running Pod. kubectl apply or kubectl edit with a new value gets rejected:

The Pod "good" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, ...

The Pod gets deleted and created again. For a Deployment you edit the Pod template, and the controller rolls out new Pods itself.

Related commands

kubectl logs <pod>
kubectl logs <pod> --previous       # log of the crashed container's previous run
kubectl describe pod <pod>          # the Command and Args blocks under Containers
kubectl exec <pod> -- ps aux        # what actually started (if the image has ps)
kubectl get pod <pod> -o jsonpath='{.spec.containers[0].command}'
kubectl explain pod.spec.containers.args

In kubectl describe pod the command looks like this:

    Command:
      sh
      -c
      echo hi; sleep 3600

ps aux inside busybox shows the process with PID 1:

PID   USER     TIME  COMMAND
    1 root      0:00 sh -c echo hi; sleep 3600

What usually breaks

• The Pod is Completed or CrashLoopBackOff a second after starting. The main process exited: a shell with no script, sh -c got the script in pieces, a command with no long-running process at the end.
• Your own command with no args, counting on the image's CMD. CMD gets dropped, the program starts without the arguments it needs.
• Your own args with no program name. If the image's ENTRYPOINT expects a program as its first argument, it gets a flag instead and crashes. Check kubectl logs --previous and docker image inspect.
• An unquoted number in args. The Pod doesn't get created, apply returns a parsing error.

What you'll need for the tasks

• Lab 1 (labs/01-ticker): command and args with sh -c, the script as one string, recreating the Pod after an edit.
• Lab 2 (labs/02-nginx-args): how args replaces the image's CMD and how to find the cause of a crash from the previous run's log.
• Commands: kubectl apply -f, kubectl get pod, kubectl logs --previous, kubectl describe pod, kubectl delete pod, docker image inspect.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-commands-and-args     # set up the starting state
./run.sh verify k8s-commands-and-args    # check the current step

In short

• command overrides the image's ENTRYPOINT, args overrides CMD.
• Only command is set: the image's CMD isn't used at all.
• Both fields are lists of strings, each element becomes a separate argument.
• sh -c expects the whole script as one string, you can't split it up.
• Images that default to a shell exit immediately: the Pod goes to Completed or CrashLoopBackOff until a long-running process gets added.
• What actually started is visible in kubectl describe pod and kubectl exec -- ps aux.