ImagePullBackOff: the Pod can't pull the image

The most common failure in Kubernetes looks boring: the Pod is created, but the container doesn't start, and the status shows ImagePullBackOff.

NAME   READY   STATUS             RESTARTS   AGE
web    0/1     ImagePullBackOff   0          90s

Kubernetes isn't at fault here. It asked for the image and didn't get it. Below, we'll look at what happens under the hood, where the real error text lives, and how to tell a typo apart from an access or network problem by that text.

How kubelet pulls the image

kubelet is the Kubernetes agent on every node that runs containers. Given a Pod spec with a specific image, it goes through several steps in order:

1. Checks whether it needs to pull the image at all. This is decided by the imagePullPolicy field (more on it below) and whether the image is already on the node.
2. If it does, it asks the container runtime (containerd, CRI-O) to pull the image.
3. The runtime parses the image name, finds the registry host through DNS, and requests the manifest and layers from the registry.
4. Once everything is downloaded, the runtime creates and starts the container.

Any step can fail. Based on the reason, kubelet sets its own state on the container, and you see it in the STATUS column.

The family of image errors

Status	What it means	Typical cause
ErrImagePull	the attempt to pull the image failed	a typo in the name or tag, no access, the registry host can't be found, no network
ImagePullBackOff	kubelet is waiting before the next attempt	the same thing as above: there have already been several attempts
InvalidImageName	the image name doesn't parse	capital letters in the repository name, Cyrillic characters, extra characters
ErrImageNeverPull	pulling is forbidden, and the node doesn't have the image	imagePullPolicy: Never and the image hasn't been loaded onto the node

There's essentially no difference between ErrImagePull and ImagePullBackOff. It's one error. After a failure, kubelet simply waits before the next attempt, and the container shows as ImagePullBackOff the whole time. The pause grows: 10 seconds, 20, 40, and so on, until it hits a ceiling of 5 minutes. So a Pod that's been sitting in ImagePullBackOff for ten minutes hasn't hung. It's waiting for the next attempt, and its STATUS toggles between the two values.

InvalidImageName and ErrImageNeverPull never touch the network at all: kubelet refuses before it even reaches out to the registry.

What the diagnostics show

kubectl describe pod web

Two parts of the output matter.

Containers → Image: what image was actually requested.

Containers:
  app:
    Image:          nginxinc/nginx-unpriviledged:1.27
    State:          Waiting
      Reason:       ImagePullBackOff

Events: what the runtime answered. This has the direct error text (lines shortened):

Events:
  Type     Reason   Message
  Normal   Pulling  Pulling image "nginxinc/nginx-unpriviledged:1.27"
  Warning  Failed   Failed to pull image "nginxinc/nginx-unpriviledged:1.27": ... pull access denied, repository does not exist or may require authorization ...
  Warning  Failed   Error: ErrImagePull
  Normal   BackOff  Back-off pulling image "nginxinc/nginx-unpriviledged:1.27"
  Warning  Failed   Error: ImagePullBackOff

From here, the error text reads almost like a reference guide:

• not found: the repository exists, but it doesn't have this tag. Almost always a typo in the tag.
• pull access denied, repository does not exist or may require authorization: this is how Docker Hub answers in two cases. Either the repository doesn't exist at all (a typo in the name), or it's private and the node has no access. Check the name first, then access.
• unauthorized or 401: another registry needs a login. You need an imagePullSecret.
• no such host: DNS can't find the registry host. Either a typo in the registry address, or the node has no DNS or internet.

If you only need the tail of events without the full describe, filtering by object helps:

kubectl get events --field-selector involvedObject.name=web --sort-by='.lastTimestamp'

Common typos

• An extra or missing letter: nginx-unpriviledged instead of nginx-unprivileged, quay.oi instead of quay.io.
• The wrong namespace in the registry: nginx/nginx-unprivileged instead of nginxinc/nginx-unprivileged.
• A tag that doesn't exist: :1.27.33 instead of :1.27.3, :v1.27.3 where tags don't use v.
• Capital letters in the repository name: Nginx:1.27.3. A repository name in a registry is written only in lowercase letters, kubelet can't parse a name like that, and the status will be InvalidImageName, not ErrImagePull.
• A Cyrillic letter instead of a Latin one, most often the Cyrillic twins of c and a. You can't tell them apart by eye, and the result is the same: InvalidImageName.

A quick way to check the theory: pull the same image by hand with docker pull, or look at its manifest with crane manifest. If the image isn't there either, the name really is wrong, and it's not the cluster's fault.

imagePullPolicy and why it affects debugging

A container has an imagePullPolicy field. If you don't set it, Kubernetes fills in a value when the Pod is created:

• Always, if no tag is given or the tag is latest;
• IfNotPresent, if the tag is specific (for example :1.27.3) or the image is set by digest.

The third value, Never, is only ever set by hand.

Always means kubelet goes to the registry every time the container starts. If the registry is unreachable, the new container won't start, even if the image is already on the node.

IfNotPresent pulls only if that image isn't on the node yet. This is more reliable, but the first run on a new node still goes to the registry.

Never means "the image is already on the node, don't go anywhere". If it's not there, the status will be ErrImageNeverPull. This mode is used in local clusters (kind, minikube), where the image is loaded by a separate command, for example kind load docker-image.

How to fix it

The field spec.containers[*].image on a Pod can be changed on a live Pod, so the name gets fixed in place:

kubectl set image pod/web app=nginxinc/nginx-unprivileged:1.27.3

Here, app is the container's name inside the Pod, not the Pod's name. You can do the same thing through an editor:

kubectl edit pod web

After the change, kubelet immediately tries to pull the new image: the pause from past failures doesn't carry over to the new name. Within a few seconds, the Pod becomes Running.

Fixing a Pod created by a Deployment this way is pointless. The Deployment creates Pods from its own spec.template, and any new Pod (restart, rollout, scaling) will come back with the old image. So you fix the template instead:

kubectl set image deployment/web app=nginxinc/nginx-unprivileged:1.27.3

The Deployment creates a new ReplicaSet, and the Pods are recreated with the fixed image. You make the permanent fix where the manifest came from: in the YAML in the repository, in the Helm chart's values. Otherwise the next deploy brings the typo back.

How to prevent it

It's convenient to check the image name before applying the manifest: docker pull or crane manifest respond in seconds. In CI, the same effect comes from a step that checks the image exists in the registry before the deploy.

A floating tag together with imagePullPolicy: Always makes every container restart depend on the registry. A specific tag and IfNotPresent remove that dependency.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start debug-image     # set up the starting state
./run.sh verify debug-image    # check the current step

In short

• ErrImagePull and ImagePullBackOff are one error, the only difference is the pause between attempts.
• The real error text is in Events from kubectl describe pod.
• not found: no such tag. pull access denied on Docker Hub: no repository or no access. no such host: DNS or the registry address.
• InvalidImageName and ErrImageNeverPull happen without ever reaching the registry.
• On a Pod, image changes on the fly; on a Deployment, you fix the template.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl get pods and kubectl describe pod <name>, the Events section.
• kubectl get pod <name> -o jsonpath='{.spec.containers[0].image}', to see the full image name.
• kubectl set image deployment/<name> <container>=<image> and kubectl rollout status deployment/<name>.