DNS inside the cluster: short name, namespace and FQDN

How resolv.conf, search domains and ndots turn wget http://web/ into a Service address, and why that same name doesn't work from a neighboring namespace.

Why this matters

Applications in a cluster almost never talk to each other by IP. They write a Service name instead: http://orders/, postgres:5432. If the name is in the wrong form, the request fails with bad address or goes to the wrong place, and the application's logs don't make the reason obvious. This lesson covers who answers these names and the rules that turn a short name into a full one.

CoreDNS: the cluster's DNS server

CoreDNS is the cluster's DNS server; it resolves Service names into their IPs. It runs as the Deployment coredns in namespace kube-system. A namespace is a logical partition of the cluster used to group objects.

In front of CoreDNS sits the Service kube-dns with a fixed ClusterIP. A ClusterIP is a Service's internal address. In kind and other kubeadm clusters it's 10.96.0.10; in k3s it's usually 10.43.0.10. Check yours:

kubectl -n kube-system get svc kube-dns

NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   72s

kubelet writes this IP into every new Pod's /etc/resolv.conf. kubelet is the Kubernetes agent on a node that runs Pods. It does this under the dnsPolicy: ClusterFirst policy, which is the Pod default. Here's the file for a Pod in namespace demo on kind:

search demo.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

The three lines of resolv.conf

nameserver

nameserver is the address a Pod sends its DNS queries to. Every query from the Pod goes to CoreDNS first. If the name isn't in the cluster zone cluster.local, CoreDNS forwards the query out, to the node's own DNS servers (the line forward . /etc/resolv.conf in its config).

search

search is the list of domains the resolver appends to short names. Order matters: the Pod's own namespace comes first.

A client writes wget http://web/. The resolver tries these in order:

1. web.demo.svc.cluster.local. Found; returns the ClusterIP.
2. If not found, web.svc.cluster.local.
3. If not found, web.cluster.local.
4. If not found, web. as is. This query goes out to external DNS.

That's why, from a Pod in demo, the short name web finds the Service web in the same namespace. From a Pod in other, the short name web finds nothing: there, the first domain is other.svc.cluster.local.

If the node's own resolv.conf has search domains, kubelet appends them to the end of the list. A kind node has none, so the example has three domains.

options ndots:5

ndots:5 tells the resolver: if the name has fewer than five dots, try the search domains first and only treat the name as absolute afterward. web.demo has one dot, so it also goes through search. The resolver first asks for web.demo.demo.svc.cluster.local. and gets no match, then web.demo.svc.cluster.local. and finds the Service.

Which names resolve

From namespace	Name	Resolves?
demo	web	yes (search appends its own namespace)
demo	web.demo	yes
demo	web.demo.svc.cluster.local	yes
other	web	no (there's no Service web in other)
other	web.demo	yes
other	web.demo.svc.cluster.local	yes

Inside your own namespace, the short name is enough. From someone else's, add <svc>.<ns>. The FQDN (the full name <svc>.<ns>.svc.cluster.local) always works, but it's a lot to type by hand.

The error for a short name from another namespace looks different depending on the setup. Once every search option has failed, the name web. goes out to external DNS. If that answers "no such name", busybox wget prints bad address 'web'. But external DNS varies. On Docker Desktop in kind, the query web. got back the special-purpose address 127.0.53.53:

wget: can't connect to remote host (127.0.53.53): Connection refused

For other short names, the same Docker Desktop DNS answers with ::, and wget prints can't connect to remote host: Connection refused with no IP. If you see Connection refused or a strange IP instead of a response from the Service, check which namespace the request is coming from first.

What DNS returns

For a regular Service, the resolved record is an A record pointing at the Service's ClusterIP. An A record maps a name to a single IPv4 address.

web.demo.svc.cluster.local. 30 IN A 10.96.112.251

The number 30 here is the TTL: how many seconds the client may cache the answer. It's set by the ttl line in CoreDNS's kubernetes plugin settings. Without that line, the plugin returns 5 seconds, but kubeadm and kind write ttl 30 into the config. Check CoreDNS's config in your cluster:

kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}'

For a Headless Service (field clusterIP: None), DNS returns the Pod IPs directly, without load balancing. This is needed for StatefulSet, the controller for stateful applications where each Pod has a stable name and address. Headless gets covered later in the track.

How to look at DNS from inside

The busybox image has the nslookup utility, which makes a DNS query. A short name resolves in the namespace where the Pod runs.

kubectl run probe -n demo --rm -it --image=busybox:1.36.1 --restart=Never -- \
  nslookup web.demo.svc.cluster.local

Server:    10.96.0.10
Address:   10.96.0.10:53

Name:   web.demo.svc.cluster.local
Address: 10.96.112.251

Server is CoreDNS itself. Address under the Name line is the Service's ClusterIP, the result of the lookup.

Why does the example use the FQDN instead of web? busybox's nslookup prints every search attempt separately, for both A and AAAA records. For a short name the output looks like an error, even though the answer is in there:

** server can't find web.cluster.local: NXDOMAIN

Name:   web.demo.svc.cluster.local
Address: 10.96.112.251

** server can't find web.svc.cluster.local: NXDOMAIN
...
command terminated with exit code 1

Look for the Name: line. If it's missing from every attempt, the name really doesn't resolve. To check whether the application itself gets through, making the actual request is more reliable: wget -qO- -T 3 http://web/.

If a suitable Pod is already running, you don't need a new one: kubectl exec makes the request right inside it.

kubectl -n demo exec probe -- wget -qO- -T 3 http://web/
kubectl -n demo exec probe -- cat /etc/resolv.conf

The second command shows that specific Pod's nameserver, search and options ndots.

What usually breaks

• A Pod reaches out by short name from another namespace. Fix: write <svc>.<ns> or the FQDN.
• A Pod manifest sets dnsPolicy: None with a dnsConfig field. Then kubelet doesn't assemble resolv.conf itself; it takes nameservers, searches and options from dnsConfig. If the manifest was copied from another namespace, short names will resolve there instead. Under the default policy, dnsConfig.searches is only appended to the end of the list.
• Old alpine images (musl instead of glibc) used to have search domain bugs. In alpine:3.20.3, short names and FQDNs resolve the same way as in busybox. If a short name doesn't work on an old image, use the FQDN.
• ndots:5 slows down external DNS. A request to api.github.com (two dots) goes through every search option first and only then goes out as is. Under heavy traffic, this gets tuned via the Pod spec's dnsConfig field (lowering ndots), or by writing external names with a trailing dot: api.github.com.

What you'll need for the tasks

• Read /etc/resolv.conf inside a Pod via kubectl exec and find its first search domain.
• Make a request from a Pod: kubectl -n <ns> exec <pod> -- wget -qO- -T 3 http://<name>/.
• Change an environment variable on a Deployment (kubectl set env or kubectl edit) and wait for the new Pods: kubectl rollout status.
• Check a Pod's DNS settings: kubectl get pod <pod> -o yaml, the dnsPolicy and dnsConfig fields.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-dns-internal     # set up the starting state
./run.sh verify k8s-dns-internal    # check the current step

In short

• Service names are resolved by CoreDNS, whose address kubelet writes into every Pod's /etc/resolv.conf.
• The search line appends <ns>.svc.cluster.local to short names, so your own namespace works without a prefix.
• From another namespace you need at least <svc>.<ns>; the FQDN always works.
• ndots:5 makes the resolver walk through the search domains before an external query, which slows down resolving public names.
• DNS answers for Service names, not for regular Pod names.