The ConfigMap is gone: bringing a service back up after a disaster

Someone ran kubectl delete configmap in the wrong context. For a while, nothing seems on fire: running Pods keep going with their old settings. But the moment Pods get recreated (a node reboot, a new release, scaling), the new ones get stuck in CreateContainerConfigError and the service goes down. This lesson is about recovery: getting the config and the traffic back. Finding who's to blame comes later.

Why this matters

Deleting a ConfigMap hits with a delay, and that's its sting. The object itself is only needed by kubelet at container creation. A container that's already running got its environment variables and doesn't look at the ConfigMap again. So the disaster usually isn't noticed at the moment of delete, but an hour or a night later, when the Pods get recreated.

By that point, nobody remembers the values that were in there. Only a backup saves you, and it has its own catch.

What CreateContainerConfigError means

It's the status of a container that kubelet couldn't assemble: it didn't have enough data to configure it. The most common scenario: a Pod references a ConfigMap or Secret through envFrom or env.valueFrom.configMapKeyRef, and the object doesn't exist. All the reference variants are covered in the "ConfigMap not found" lesson.

In the Pod's description, it looks like this:

kubectl describe pod api-xxx

In the Events section:

Warning  Failed  10s (x5 over 30s)  kubelet  Error: configmap "api-config" not found

If the same ConfigMap is mounted as a volume (volumes.configMap), the status is different: the Pod sits in ContainerCreating, and the Events show a FailedMount event. kubelet mounts volumes before it assembles environment variables.

Importantly, the Pod doesn't go into CrashLoopBackOff: it never even started. CrashLoopBackOff means "the container started and crashed", CreateContainerConfigError means "the container never got created at all". The difference matters: there are no application logs to check, you have to look at Events.

Check that the object is really gone:

kubectl get configmap

How to restore a ConfigMap

The source of the values comes in two flavors. If the config is simple and the values are known from the service's docs, build it by hand:

kubectl create configmap api-config \
  --from-literal=API_PORT=8080 \
  --from-literal=API_HOST=0.0.0.0 \
  --from-literal=LOG_LEVEL=info

If a backup was taken ahead of time, apply it:

kubectl apply -f backup/api-config.yaml

There's a trap here. kubectl get configmap api-config -o yaml returns the object along with internal fields the cluster filled in:

metadata:
  creationTimestamp: "2026-10-12T10:00:00Z"
  name: api-config
  namespace: default
  resourceVersion: "1234"
  uid: 6f1c2a3e-...

resourceVersion is the object's version number in the cluster's storage. While the object doesn't exist, a dump like this applies fine: kubectl resets resourceVersion on create, and the cluster issues a new uid. Problems start when the object already exists. For example, someone already managed to recreate the ConfigMap by hand. Then the resourceVersion from the file doesn't match the current one, and the API server refuses:

Error from server (Conflict): error when applying patch:
...
Operation cannot be fulfilled on configmaps "api-config": the object has been modified; please apply your changes to the latest version and try again

So strip the internal fields from the dump: uid, resourceVersion, creationTimestamp, and managedFields too if it ended up in the file. Deleting these lines with a text editor is enough. Keep the namespace field: it says where to restore to.

The object will appear, but that still doesn't guarantee a working service.

When the Pods come back up

kubelet reads the ConfigMap in three situations:

1. At container creation. It reads the object once and builds the environment variables or mounts the volume. No ConfigMap, no container.
2. On a mounted volume update. kubelet updates the files with a delay: the sync period (1 minute by default) plus cache delay. Works for volumes, but not for envFrom and not for a volume with subPath.
3. On container restart. A new start reads the ConfigMap again.

Pods stuck in CreateContainerConfigError never got a container created in the first place. kubelet keeps retrying on its own, and once the ConfigMap appears, the next attempt will succeed. You don't have to recreate these Pods, but waiting for the next attempt sometimes takes longer than you'd like.

Speed it up by deleting the Pods: the ReplicaSet will create new ones right away, and they'll see the ConfigMap at startup.

kubectl delete pods -l app=api

Or go through the Deployment, which recreates Pods one at a time:

kubectl rollout restart deployment/api

Nothing to lose: the stuck Pods weren't working anyway.

envFrom and volumes after restoring

Reference type	What happens after the ConfigMap is restored
envFrom	values are read at container start; running Pods need a restart
volume + volumeMounts	kubelet updates the files on its own, with a delay of up to a minute or more
volume with subPath	doesn't update, needs a new container

envFrom:
  - configMapRef:
      name: api-config

All the ConfigMap's keys become environment variables at container start. If the ConfigMap was restored with different values than before, running Pods won't see them; they need a restart.

volumes:
  - name: config
    configMap:
      name: api-config
volumeMounts:
  - name: config
    mountPath: /etc/api

Here the ConfigMap is mounted as files. kubelet updates them on its own, but the app has to be able to reread the config: on a SIGHUP signal, or by watching the file.

For recovery, the first option is the more annoying one. Pods that survived the deletion keep running with the old env, and if the backup differs from what was there, some replicas keep working the old way until they get restarted.

How not to step on this again

Critical ConfigMaps get dumped to a file regularly. The dump is cleaned of internal fields right away, so that on the day of the disaster it applies in any situation, including over an object someone already managed to recreate. yq (mikefarah/yq version 4) makes this convenient:

kubectl get configmap api-config -o yaml \
  | yq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields)' \
  > backup/api-config.yaml

There's also the kubectl-neat plugin, which strips internal fields entirely, installed via krew.

In production, this role is usually covered by GitOps: ConfigMaps live in a repository, and the cluster pulls them through Argo CD or Flux. Then recovery means re-syncing, not a manual apply from memory. Permissions for delete in production are granted separately from permissions for apply, and before dangerous commands, check the context: kubectl config current-context.

Common mistakes

Expecting the service to go down right after the delete. Running Pods survive the ConfigMap's deletion, and the problem surfaces later, at the first recreation. Silence after delete doesn't mean everything is fine.

Applying a raw dump over a live object. kubectl get -o yaml without cleaning gives a file with a stale resourceVersion. If the object got recreated in the meantime, apply fails with Conflict.

Restoring values from memory. The ConfigMap appears, the Pods come up, but with different settings. When the correct config is later restored, Pods with envFrom won't see it without a restart. If a backup exists, take the values from it right away.

Deleting Pods before restoring the ConfigMap. The new Pods will hit the same error. Object first, then Pods.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-configmap-disaster     # set up the starting state
./run.sh verify k8s-configmap-disaster    # check the current step

In short

• CreateContainerConfigError means the container was never created; there are no application logs, read the Events.
• The cause is most often a missing ConfigMap or Secret from envFrom or configMapKeyRef. A missing volume gives ContainerCreating and FailedMount.
• Deleting a ConfigMap doesn't touch running Pods, it breaks new ones.
• A dump from kubectl get -o yaml gets cleaned of uid, resourceVersion, creationTimestamp: with a stale resourceVersion it won't apply over a recreated object.
• After restoring, stuck Pods come up on their own; deleting the Pods or rollout restart speeds it up.
• A backup of critical ConfigMaps, or a GitOps repository, turns this kind of disaster into a single command.

What you'll need for the tasks

• kubectl config current-context, to make sure you're working in the lab's cluster.
• kubectl get pods, kubectl describe pod <name>, the Events section.
• kubectl get configmap, kubectl get configmap <name> -o yaml.
• kubectl apply -f <file> and a text editor to edit the YAML.
• kubectl rollout restart deployment/<name>, kubectl rollout status deployment/<name>, kubectl logs.

Useful commands

kubectl get pods
kubectl describe pod <pod-name>
kubectl get configmap
kubectl rollout status deployment/api --timeout=120s
kubectl get events --sort-by=.lastTimestamp | tail

kubectl explain pod.spec.containers.envFrom
kubectl explain configmap.data