Headless Service: clusterIP None

What changes when a Service hands the client a list of Pod addresses instead of one virtual IP.

Why this matters

A regular Service hides its Pods behind one address, and the client doesn't care how many there are. Sometimes the client needs the opposite: to know every Pod. A gRPC client wants to spread connections itself, a database looks for its peers at startup, a StatefulSet replica wants a stable name. That's what a Headless Service is for, and it has its own traps: there are no addresses in DNS until the Pods are ready, and you can't change clusterIP on a live Service.

Regular ClusterIP vs. headless

A regular Service:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - name: http
      port: 80
      targetPort: 8080

The cluster assigns it one virtual IP (spec.clusterIP). DNS resolves web.<ns>.svc.cluster.local to that one IP. From there, kube-proxy on the node redirects the traffic through iptables/IPVS to one of the ready Pods.

A Headless Service differs by one line:

apiVersion: v1
kind: Service
metadata:
  name: web-hl
spec:
  clusterIP: None
  selector:
    app: web
  ports:
    - name: http
      port: 80
      targetPort: 8080

clusterIP: None means "don't assign a virtual IP". CoreDNS now returns not one IP, but one A record per ready Pod. kube-proxy doesn't serve this Service at all: the client connects straight to the Pod's IP, on whatever port the container listens on.

The difference shows right away in kubectl get svc output: a headless Service has None in the CLUSTER-IP column:

kubectl get svc web web-hl

NAME     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
web      ClusterIP   10.96.190.51   <none>        80/TCP    67s
web-hl   ClusterIP   None           <none>        80/TCP    64s

	Regular ClusterIP	Headless (clusterIP: None)
Virtual IP	assigned	none
What DNS returns	one A record for the ClusterIP	one A record per ready Pod
kube-proxy in the path	yes	no
Who picks the Pod	kube-proxy	the client
Names for individual Pods	none	for a StatefulSet, and for a Pod with hostname + subdomain

What DNS returns

A query for web-hl.demo.svc.cluster.local from a Pod in kind:

Name:	web-hl.demo.svc.cluster.local
Address: 10.244.0.6
Name:	web-hl.demo.svc.cluster.local
Address: 10.244.0.5
Name:	web-hl.demo.svc.cluster.local
Address: 10.244.0.8

That's the IP of every ready Pod. The same query for the regular web returns a single line with the ClusterIP 10.96.190.51.

The order of the addresses changes from query to query. CoreDNS's loadbalance plugin shuffles it (it's on by default in kind and kubeadm). Many clients take the first address from the answer, so the load spreads out somewhat. But you can't call that load balancing: a client will cache the answer and keep hitting the same Pod. Distributing connections is the client's own job.

Only ready Pods make it into the answer. If a Pod fails its readinessProbe, the controller marks its address ready: false in the EndpointSlice, and CoreDNS stops returning it. If a Pod disappears entirely, its address drops out of the next answer too. DNS has no health check of its own; it sees exactly what's recorded in the EndpointSlice.

Sometimes you need the addresses before the Pods are ready. For example, database nodes look for each other over DNS at startup and only consider themselves ready once they've all found each other. For this, a Service has a field called publishNotReadyAddresses: true: with it set, DNS returns the addresses of every Pod under the selector, ready or not.

SRV records

For every named port on a Service (ports[].name), CoreDNS creates an SRV record shaped like:

_<port name>._<protocol>.<service>.<ns>.svc.cluster.local

For example, _http._tcp.web-hl.demo.svc.cluster.local. An SRV record holds a hostname and a port. A regular Service has one too, but just a single one: the Service's name and port (80). A headless Service has one SRV record per ready Pod, with that Pod's DNS name and targetPort (8080):

_http._tcp.web.demo.svc.cluster.local	service = 0 100 80 web.demo.svc.cluster.local
_http._tcp.web-hl.demo.svc.cluster.local	service = 0 33 8080 10-244-0-8.web-hl.demo.svc.cluster.local
_http._tcp.web-hl.demo.svc.cluster.local	service = 0 33 8080 10-244-0-6.web-hl.demo.svc.cluster.local
_http._tcp.web-hl.demo.svc.cluster.local	service = 0 33 8080 10-244-0-5.web-hl.demo.svc.cluster.local

Clients that care who's listening on which port need this, Kafka brokers for example.

Names for individual Pods

This part trips people up. A Headless Service by itself doesn't give a Pod a name like web-0.web-hl.demo.svc.cluster.local. A Pod's name only shows up in DNS once the Pod has its hostname and subdomain fields set, and subdomain matches the name of a Headless Service in the same namespace.

Almost nobody sets these fields by hand. A StatefulSet fills them in: it's the controller for stateful applications, where every Pod gets a stable name and address. It sets hostname to the Pod's name (db-0, db-1) and takes subdomain from its own spec.serviceName field. If no Service with that name exists, there are no Pod names in DNS either.

Pods from a Deployment (web-79cbd46fb-f98f8) don't have these fields. CoreDNS gives them names by IP instead: 10-244-0-6.web-hl.demo.svc.cluster.local, visible in the SRV answer above. They're not very useful, since the IP changes whenever the Pod is recreated. A name like web-a.web-hl... for such a Pod returns NXDOMAIN.

Per the Kubernetes DNS spec, Pod names are only guaranteed for a Headless Service. CoreDNS in kind also answers db-0.db when Service db is a regular one, but don't rely on that: it depends on the DNS implementation.

Why this matters

Three typical scenarios:

• A gRPC client with client-side load balancing resolves the name, gets all the IPs, and distributes connections across the pool itself. kube-proxy balances at the TCP connection level, but gRPC pushes many requests over one connection. With a regular Service, all of the client's load ends up on one Pod.
• Kafka, RabbitMQ, and Elasticsearch look for their peers over DNS at startup. Headless hands them the list, the node sees its peers and joins the cluster.
• NAT through kube-proxy sometimes breaks sticky sessions or metrics (the source address changes). Headless removes kube-proxy from the path.

Ordinary web applications don't need headless. If you want balancing done by the cluster, keep a regular Service.

Endpoints and EndpointSlice

Addresses for a Headless Service are collected the same way as for a regular one. The EndpointSlice controller creates EndpointSlice objects with the Pods' IPs and their readiness, and a separate Endpoints controller mirrors the list into the old Endpoints object. CoreDNS reads the EndpointSlice and builds its answer from it.

To see who the Service sees:

kubectl get endpointslices -l kubernetes.io/service-name=web-hl

NAME           ADDRESSTYPE   PORTS   ENDPOINTS                                      AGE
web-hl-mm5m5   IPv4          8080    10.244.0.8,10.244.0.6,10.244.0.5 + 1 more...   86s

The ENDPOINTS column lists every address, including ones that aren't ready. Each one's readiness shows up in kubectl describe endpointslice -l kubernetes.io/service-name=web-hl, in the Conditions field.

The old command still works, but prints a warning: v1 Endpoints has been deprecated since Kubernetes 1.33.

kubectl get endpoints web-hl

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME     ENDPOINTS                                         AGE
web-hl   10.244.0.5:8080,10.244.0.6:8080,10.244.0.8:8080   86s

If there are no addresses at all, the selector doesn't match the Pods' labels. If there are addresses but DNS isn't returning them, the Pods aren't ready.

How to check the resolution

You need a Pod with nslookup; busybox is the easiest choice. Resolve from inside the cluster: CoreDNS isn't reachable from outside.

kubectl run dns-test --rm -it --image=busybox:1.36.1 --restart=Never -- \
  nslookup web-hl.demo.svc.cluster.local

--rm deletes the Pod after it exits. The example uses the full name (FQDN): with the short name web-hl, busybox prints an NXDOMAIN refusal for every search domain and exits with code 1, even though the addresses you need are right there in the output. The lesson on cluster DNS explains why.

Query SRV records with the -type=srv flag:

kubectl run dns-test --rm -it --image=busybox:1.36.1 --restart=Never -- \
  nslookup -type=srv _http._tcp.web-hl.demo.svc.cluster.local

Common mistakes

• Turning a regular Service into headless with kubectl apply or edit. The clusterIP field can't change once the Service is created; the API answers spec.clusterIPs[0]: Invalid value: ["None"]: may not change once set. You have to delete the Service and create it again.
• Expecting DNS to have a Deployment Pod's name (web-a.web-hl...). It doesn't; only the by-IP name exists. Stable names come from a StatefulSet or from a Pod's hostname and subdomain fields.
• A StatefulSet's spec.serviceName points at a Service that doesn't exist, or at a different one. The Pods start, but their names don't resolve.
• The Pods aren't ready, and DNS for the headless Service returns an empty answer. Check Pod readiness and the EndpointSlice first, then DNS.
• Checking with nslookup from busybox by the short name and panicking at NXDOMAIN. Check with the FQDN, or query with wget instead.

What you'll need for the tasks

• DNS query from a Pod: kubectl exec <pod> -- nslookup <name>.<ns>.svc.cluster.local, add -type=srv for SRV.
• Readiness of a Service's addresses: kubectl describe endpointslice -l kubernetes.io/service-name=<svc>.
• Service fields: kubectl get svc <svc> -o yaml, especially clusterIP and publishNotReadyAddresses.
• Changing a Service field: kubectl edit svc <svc> or kubectl patch svc <svc> -p '{"spec":{...}}'.
• StatefulSet Pod status and why it isn't ready: kubectl get pod, kubectl describe pod <pod>, the Events section.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-headless-service     # set up the starting state
./run.sh verify k8s-headless-service    # check the current step

In short

• A Headless Service is clusterIP: None; no virtual IP gets assigned.
• DNS returns one A record per ready Pod, and kube-proxy drops out of the path.
• CoreDNS shuffles the order of the addresses, but distributing load is still the client's job.
• Pods that aren't ready don't show up in DNS unless the Service has publishNotReadyAddresses: true.
• Names like db-0.db-hl... come from a StatefulSet's spec.serviceName or from a Pod's hostname and subdomain fields, not from headless itself.
• Needed for gRPC with client-side load balancing, and for clustered systems that look for their peers over DNS.