DNS doesn't resolve: when CoreDNS is down

The incident shows up as "everything is broken". Apps can't find the database, the queue can't find the broker, health checks fail on half the services. Meanwhile the Pods are alive, the network between nodes works, and kubectl answers normally.

When the list of affected services is too wide, there's usually one cause: the cluster's DNS infrastructure. In this lesson you'll learn to tell a cluster-wide DNS outage apart from a single Service problem, and walk the check chain that shows exactly where it broke.

Concepts

• CoreDNS — the cluster's DNS server. Runs as the Deployment coredns in namespace kube-system.
• Service kube-dns — the Service in front of the CoreDNS Pods, with a fixed ClusterIP. The name is historical, left over from the old kube-dns.
• Corefile — CoreDNS's config. Lives in ConfigMap coredns in kube-system.
• EndpointSlice — the list of Pod addresses a Service sends traffic to. An empty list means traffic has nowhere to go.

How DNS works inside the cluster

CoreDNS runs as an ordinary Deployment. In clusters created by kubeadm (and in kind, which the course labs run on), it has two replicas by default; in k3s, one. In front of the Pods sits Service kube-dns with a fixed ClusterIP. kubelet writes this address into every new Pod's /etc/resolv.conf. Here's what the file looks like in a Pod from namespace default in kind:

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

nameserver is Service kube-dns's ClusterIP. In kind and kubeadm it's 10.96.0.10, in k3s it's 10.43.0.10. search lists the suffixes the resolver appends to incomplete names. ndots:5 means: if a name has fewer than five dots, the resolver first tries it with the suffixes appended, and only then asks for the name as is.

When curl webapp runs inside a Pod, here's what happens:

1. the resolver appends a suffix: webapp.default.svc.cluster.local;
2. it sends a UDP request to 10.96.0.10:53;
3. rules that kube-proxy wrote into the node's kernel ahead of time route the packet to one of the CoreDNS Pods;
4. CoreDNS replies with Service webapp's ClusterIP. It doesn't look up the Service in the API on every request: the kubernetes plugin keeps a copy of Services and EndpointSlices in memory and updates it through a watch.

If any link in this chain breaks, resolution goes down for the whole cluster at once.

Diagnosis: nslookup from a Pod

Command number one. Bring up a temporary Pod and ask DNS for a name that exists in any cluster, Service kubernetes in namespace default:

kubectl run dns-probe --image=busybox:1.36.1 --restart=Never -- sleep 3600

kubectl exec dns-probe -- nslookup kubernetes.default.svc.cluster.local

A healthy answer in kind:

Server:		10.96.0.10
Address:	10.96.0.10:53

Name:	kubernetes.default.svc.cluster.local
Address: 10.96.0.1

Instructions often give the short form, nslookup kubernetes.default. Busybox sends that name both with the search suffixes and as is. The "as is" query gets answered by an external DNS server, and on some machines it returns something of its own. Busybox prints that answer, and the output is confusing. The full name doesn't leave that ambiguity.

What you get instead of a healthy answer:

Symptom in busybox nslookup output	What it means
nslookup: write to '10.96.0.10': Connection refused and ;; connection timed out; no servers could be reached	Service kube-dns has no ready addresses at all: there are no CoreDNS Pods, they aren't Ready, or the Service doesn't select them
;; connection timed out; no servers could be reached without Connection refused	kube-dns has ready addresses, but CoreDNS doesn't respond: the Pods just crashed, are overloaded, or a NetworkPolicy is dropping packets
** server can't find <name>: NXDOMAIN for every suffix, while kubernetes.default.svc.cluster.local resolves fine	DNS works, that name just doesn't exist: a typo in the Service name, or the wrong namespace

The first two lines point to infrastructure, the last one to a specific name. These are different incidents. An empty address list on a Service (say, a selector that doesn't match the Pod's labels) has no effect on NXDOMAIN: a ClusterIP Service resolves even without Pods behind it.

Where CoreDNS lives

kubectl get deploy coredns -n kube-system

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
coredns   2/2     2            2           30d

Pod status:

kubectl get pods -n kube-system -l k8s-app=kube-dns

NAME                       READY   STATUS    RESTARTS   AGE
coredns-7d764666f9-mmp4x   1/1     Running   0          30d
coredns-7d764666f9-nfg59   1/1     Running   0          30d

READY 0/1, CrashLoopBackOff, or no Pods at all mean DNS is down. Scaling to zero looks the same. Service kube-dns stays in place with its ClusterIP, but there's nothing behind it.

Check the address list behind the Service through EndpointSlice:

kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns

NAME             ADDRESSTYPE   PORTS     ENDPOINTS   AGE
kube-dns-bll6p   IPv4          <unset>   <unset>     79s

<unset> in the ENDPOINTS column means there are no addresses at all: either there are no Pods, or the Service doesn't select them. Keep in mind this column also shows not-ready addresses. If the CoreDNS Pods are crashing in CrashLoopBackOff, their IPs stay listed there, just marked ready: false, and traffic won't go to them (conditions in EndpointSlice). You can see each address's readiness like this:

kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]} ready={.conditions.ready}{"\n"}{end}'

The older kubectl get endpoints kube-dns -n kube-system command shows only ready addresses, and prints <none> when there are none. But the v1 Endpoints object is deprecated as of Kubernetes 1.33, and kubectl prints a warning about it.

No ready addresses on kube-dns means traffic has nowhere to go. Next you need to figure out why. There are three possibilities: zero replicas, Pods exist but aren't Ready, or the Pods are Ready but the Service doesn't select them. Check the last case the same way as in the selector lesson: put the Service's selector next to the Pods' labels.

Inside the Corefile

CoreDNS's config lives in ConfigMap coredns in kube-system:

kubectl get cm coredns -n kube-system -o yaml

The key lines: kubernetes cluster.local answers names under *.cluster.local, forward . /etc/resolv.conf sends external domains on to the node's DNS server.

CoreDNS finds an error in the Corefile while parsing it. What happens next depends on when:

• At process startup. CoreDNS writes the error to the log and exits, and the Pod goes into CrashLoopBackOff.
• In a running Pod. The reload plugin checks the Corefile against the previous version every 30 seconds (plus a random jitter of up to 15 seconds). It doesn't apply a broken version: it writes the error to the log and keeps running with the old config.

The second case is sneakier. The Corefile edit goes through "with no consequences", DNS still answers, everyone moves on. Then a day later the CoreDNS Pods get recreated (a node restart, an upgrade, an eviction), they read the broken file from scratch, and DNS goes down across the whole cluster. The cause is visible in the logs:

kubectl logs -n kube-system -l k8s-app=kube-dns

If the container just restarted and the current log is empty, add --previous: kubectl will then show the output of the previous, crashed run.

Check the Corefile when Pods exist but keep crashing or aren't Ready, or when replicas are Ready but external names don't resolve. If replicas are at zero, the config isn't the issue.

Fixes

Problem	What to do
zero replicas	kubectl scale deploy coredns -n kube-system --replicas=2
Pod in CrashLoopBackOff	read the cause: kubectl logs -n kube-system -l k8s-app=kube-dns --previous
broken Corefile	kubectl edit cm coredns -n kube-system, fix it, then kubectl -n kube-system rollout restart deployment/coredns
Pod Ready, EndpointSlice empty	compare Service kube-dns's selector with the CoreDNS Pods' labels

After you fix the ConfigMap, Pods in CrashLoopBackOff will come back up on their own: kubelet updates the file inside the Pod, and the next restart reads the fixed config. But both paths come with a delay. The ConfigMap file takes about a minute to reach the Pod (docs), and the pause between restarts doubles after every crash, up to five minutes (docs). If the Pods had been crashing for a while, the wait will be long too. rollout restart recreates the Pods right away, and they read the fixed config immediately.

After the fix, check resolution the same way, with nslookup from a test Pod. You usually don't need to restart your apps: libc's resolver hits DNS on every new request. The exception is programs with their own DNS cache. nginx, for example, resolves the name from proxy_pass once, when it loads its config (docs), and if it started while DNS was down, you'll need to restart it.

Common mistakes

• Starting with the application. If even kubernetes.default.svc.cluster.local doesn't resolve, it's not about one Service.
• Stopping at kubectl get pods: "the CoreDNS Pod is Running, so DNS is fine." Traffic goes to the Service's ClusterIP, and without ready addresses in the EndpointSlice it never reaches the Pod.
• Deleting and recreating Service kube-dns. A new Service without an explicit clusterIP gets a different address, while every Pod's /etc/resolv.conf still has the old one written in.
• Fixing the Corefile by cutting out plugins. Without forward, external names stop resolving; without kubernetes, Service names do.
• Editing the Corefile and not checking CoreDNS's logs afterward. The reload error shows up right away, the crash comes later.

How to avoid this again

Keep CoreDNS at a minimum of two replicas on different nodes (podAntiAffinity or topologySpreadConstraints). Alert on kube-dns having empty endpoints, not on "cluster availability": empty DNS endpoints are caught in seconds, while app degradation spreads out over minutes. Ship Corefile edits through a GitOps repository, so a broken config has something to roll back to.

The check order as a single list

# How many replicas does the Deployment want, and how many are ready?
kubectl get deploy coredns -n kube-system

# Do CoreDNS Pods exist, and are they Ready?
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Does Service kube-dns have addresses?
kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns

# Does DNS respond from a Pod?
kubectl exec dns-probe -- nslookup kubernetes.default.svc.cluster.local

The chain: deploy → pods → endpoints → nslookup. Wherever it breaks, that's where the cause is.

In short

• A mass "everything is broken" failure with live Pods: suspect DNS.
• nslookup kubernetes.default.svc.cluster.local from a temporary Pod separates an infrastructure failure from a typo in a Service name.
• No ready addresses on kube-dns: either there are no replicas, they aren't Ready, or the Service doesn't select them.
• A broken Corefile doesn't bring down a running CoreDNS, it blows up on the next Pod recreation. The cause is in the logs, sometimes with --previous.
• Two replicas and an alert on kube-dns endpoints close off most of these incidents.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl exec dns-probe -- nslookup kubernetes.default.svc.cluster.local.
• The chain kubectl get deploy, kubectl get pods -l k8s-app=kube-dns, kubectl get endpointslice -l kubernetes.io/service-name=kube-dns, all with -n kube-system.
• kubectl logs -n kube-system -l k8s-app=kube-dns and the --previous flag.
• kubectl get cm coredns -n kube-system -o yaml and kubectl edit cm.
• kubectl -n kube-system rollout restart deployment/coredns.
• kubectl get svc <svc> -n kube-system -o jsonpath='{.spec.selector}', kubectl get pods -n kube-system --show-labels, and kubectl patch svc.