ConfigMap not found

What CreateContainerConfigError means, which objects a Pod references by name, and why a stuck Pod comes up on its own once the ConfigMap exists.

Why this matters

The image downloaded, the node was found, and the container still won't start:

NAME     READY   STATUS                       RESTARTS   AGE
webapp   0/1     CreateContainerConfigError   0          45s

The status looks long and alarming, but it means something simple: the Pod references an object that doesn't exist in the cluster. Usually that's a ConfigMap, an object holding configuration as key-value pairs, or a Secret.

CreateContainerConfigError and its neighbors

The status looks like ImagePullBackOff, but a different step is failing. ImagePullBackOff is about the image itself. CreateContainerConfigError means kubelet couldn't build the container's configuration: environment variables, mount points. The most common cause: an environment variable references a ConfigMap, Secret, or key in them that doesn't exist.

STATUS	At what step	Typical cause
ImagePullBackOff	downloading the image	a typo in the name or tag, no access to the registry
CreateContainerConfigError	building the container's configuration	env/envFrom references a ConfigMap or Secret that doesn't exist, or a key that isn't in it
CreateContainerError	creating the container in the runtime	the runtime refused to create the container
RunContainerError	starting the process	for example, a file from command isn't in the image

All four show up as the container state Waiting, only the reason differs. The word Config in the reason is a hint to look for references to external objects in the spec.

If the process started and immediately crashed, the status will be different: CrashLoopBackOff. That's an application error, not a Pod configuration error.

Where a Pod can reference objects

The spec has several places where a reference is made by name:

spec:
  containers:
    - name: app
      image: ...
      env:
        - name: DB_URL
          valueFrom:
            configMapKeyRef:
              name: db-config        # ← ConfigMap and its key url
              key: url
        - name: API_TOKEN
          valueFrom:
            secretKeyRef:
              name: api-secret       # ← Secret and its key token
              key: token
      envFrom:
        - configMapRef:
            name: app-config         # ← the whole ConfigMap
        - secretRef:
            name: db-creds           # ← the whole Secret
      volumeMounts:
        - name: static
          mountPath: /static
  volumes:
    - name: static
      configMap:                     # ← ConfigMap as files
        name: static-files
    - name: certs
      secret:                        # ← Secret as files
        secretName: tls-cert
    - name: storage
      persistentVolumeClaim:         # ← PVC
        claimName: app-data

Any of these references will keep the container from starting if the object doesn't exist. But the status in kubectl get pods will differ:

What's missing	STATUS	Where to look for the error text
ConfigMap or Secret from env/envFrom	CreateContainerConfigError	Events, reason Failed
a key in an existing ConfigMap or Secret (configMapKeyRef, secretKeyRef)	CreateContainerConfigError	Events, reason Failed
ConfigMap or Secret from volumes	ContainerCreating	Events, reason FailedMount
PVC	Pending	Events, reason FailedScheduling

kubelet checks volume references earlier, when it mounts the volumes. The Pod sits in ContainerCreating with no explicit error in STATUS, and the reason only shows up in Events. The scheduler also checks the PVC: without a PVC the Pod won't even get a node.

What diagnostics show

kubectl describe pod webapp

In the State section:

State:          Waiting
  Reason:       CreateContainerConfigError

In Events:

Events:
  Type     Reason   Message
  Normal   Pulled   Successfully pulled image ...
  Warning  Failed   Error: configmap "app-config" not found

If the ConfigMap exists but doesn't have the needed key, the message is different:

Warning  Failed   Error: couldn't find key url in ConfigMap default/db-config

One line with the object's name is enough to fix it. If there are several broken references, kubelet reports only the first one. It checks envFrom first, then env in order. Fix one reference, and kubelet moves to the next, showing a new error in Events. The container won't start until all required objects are found.

Two more commands that speed up the diagnosis:

kubectl get pod webapp -o jsonpath='{.spec.containers[0].envFrom}'

The first shows what the Pod references through envFrom. For env, use .env instead of .envFrom. The second shows what actually exists in the namespace:

kubectl get configmap

optional, so it doesn't fail

A reference can be marked optional:

envFrom:
  - configMapRef:
      name: app-config
      optional: true

Then, if the ConfigMap is missing, the container starts without those variables. optional is available on every reference type listed above except PVC. It fits configuration that may or may not be there.

By default a reference is required. The Pod will wait for the object, and that's the right behavior for required settings. Starting silently without a database password is worse than not starting at all.

When the container sees the new ConfigMap

It depends on how it's referenced:

Method	When the container sees changes
env / envFrom	only at container startup, values are fixed after that
volume with configMap	kubelet updates the files on its own, delayed by up to the kubelet sync period (1 minute by default) plus cache delay
volume with subPath	doesn't update, needs a new container

That leads to a non-obvious consequence. With envFrom, the variables are fixed at startup, but a Pod in CreateContainerConfigError has never started its container yet. kubelet keeps retrying, and as soon as the ConfigMap shows up, the next attempt succeeds. You don't need to recreate the Pod.

It's a different story if the Pod has been running for a while and the ConfigMap changes. Its environment variables stay old, so you need a new container. For a Deployment, one command does this, recreating its Pods one at a time:

kubectl rollout restart deployment/webapp

A standalone Pod, without a Deployment, has to be deleted and created again.

How it's fixed

A missing ConfigMap is created imperatively:

kubectl create configmap app-config --from-literal=MODE=production

Several keys:

kubectl create configmap app-config \
  --from-literal=MODE=production \
  --from-literal=LOG_LEVEL=info

From a file, the filename becomes the key:

kubectl create configmap app-config --from-file=config.json

A Secret is created the same way, with kubectl create secret generic. A few seconds later, the Pod moves to Running on its own.

If the object exists but is named differently, fix whichever side made the mistake. A typo in the Pod or Deployment manifest is fixed in the manifest. If whoever created the ConfigMap got it wrong, create the object under the correct name.

Common mistakes

Deleting the Pod to "restart" it. A Pod in CreateContainerConfigError is already being restarted by kubelet on its own. Deleting it does nothing while the object is missing, and a Pod from a Deployment comes back with the same error.

Setting optional: true just to get the Pod running. The Pod starts, but without the settings. The error moves from kubectl get pods into the application's logs, and it's harder to track down there.

Editing the ConfigMap and waiting for new values in env. Environment variables are only read at container startup. Files from a volume update, variables don't.

How to prevent it

ConfigMaps and Secrets are applied before the Pod that references them. In Kustomize and Helm this is handled by resource ordering. With a plain kubectl apply -f ./manifests/, files are applied in alphabetical order, so configuration goes into a file that comes earlier. Even if the order breaks, the Pod will wait for the object and come up on its own.

For optional configuration, set optional: true explicitly. For required configuration, leave the default so the Pod doesn't come up with empty settings.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start debug-config     # set up the starting state
./run.sh verify debug-config    # check the current step

In short

• CreateContainerConfigError means env or envFrom references a missing ConfigMap, Secret, or key in them.
• References from volumes give ContainerCreating with FailedMount, a PVC gives Pending.
• kubelet writes the missing object's name directly in Events, one error at a time.
• optional: true allows starting without the object, by default the Pod waits.
• After the ConfigMap is created, a stuck Pod comes up on its own, but a running Pod won't see new values from env/envFrom without a restart.

What you'll need for the tasks

• kubectl config current-context, to make sure you're working in the lab cluster.
• kubectl get pods, kubectl describe pod <name>, the Events section.
• kubectl get configmap, kubectl get secret, kubectl get configmap <name> -o yaml.
• kubectl create configmap ... --from-literal=... and kubectl create secret generic ... --from-literal=....
• kubectl logs, kubectl exec ... -- printenv.
• kubectl rollout restart deployment/<name> and kubectl rollout status.