Endpoints and selector: how a Service loses a Pod

Where a Service gets its list of addresses, why Endpoints gave way to EndpointSlice, and what knocks a Pod out of that list.

Why this matters

A Service receives a request and sends it to one of its Pods. But the Service itself doesn't know where those Pods are. The list of addresses lives in a separate object, and when a Service "doesn't respond," that list is almost always the culprit: it's empty, it has the wrong Pods, or the addresses are marked not ready. If you know how to read it, you find the cause in a minute instead of restarting everything in turn.

What Endpoints is

A Service holds a selector. For example:

spec:
  selector:
    app: api

For every Service with a selector, kube-controller-manager automatically creates an object listing the addresses of the Pods that match the selector. Next to each address it records whether the Pod is ready to take traffic. Historically that object was called Endpoints; it's now being replaced by EndpointSlice. Both are created automatically and show the same picture.

kubectl get svc api
kubectl get endpoints api
kubectl get endpointslice -l kubernetes.io/service-name=api

EndpointSlice instead of Endpoints

The Endpoints object (API group v1) has been deprecated since Kubernetes 1.33. It's still created, but kubectl prints a warning on every request:

kubectl get endpoints api

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME   ENDPOINTS                         AGE
api    10.244.0.7:8080,10.244.0.9:8080   1s

The replacement is EndpointSlice (discovery.k8s.io/v1). This is what kube-proxy reads when it sets up rules for traffic. kube-proxy is the component on every node that turns a Service and its addresses into rules on that node's network.

The reason for the switch is scale. Endpoints is one object per Service, with all addresses in a single list. For a Service with a thousand Pods, any change (a Pod comes up, a Pod goes down) rewrites the whole object, and every node in the cluster gets the new version. On large clusters that's a noticeable load on the control plane and the network.

EndpointSlice cuts that same list into slices, up to 100 addresses in each by default. One Pod changes, and only its slice gets rewritten. That's where the name comes from: a slice is a piece of the full list.

kubectl get endpointslice -l kubernetes.io/service-name=api

NAME        ADDRESSTYPE   PORTS   ENDPOINTS                          AGE
api-pcn6v   IPv4          8080    10.244.0.5,10.244.0.9,10.244.0.7   2s

A slice's name is generated automatically (api-pcn6v), so slices are usually looked up by the label kubernetes.io/service-name, not by the Service's name.

Notice: the EndpointSlice shows three addresses, while Endpoints above shows two. The third Pod isn't ready. Endpoints hides such addresses in a separate field, while EndpointSlice keeps them in the main list with a flag. More on this in "What kubectl shows you."

What else EndpointSlice can do that Endpoints couldn't:

• addressType: IPv4, IPv6 and FQDN addresses live in separate slices, which is what dual-stack Services rely on;
• per-address conditions: ready, serving, terminating. Endpoints only knows "ready or not";
• topology hints for routing within your own zone.

	v1 Endpoints	discovery.k8s.io/v1 EndpointSlice
Status	deprecated since 1.33	current
Objects per Service	one	one slice per 100 addresses
One Pod changes	the whole list gets rewritten	one slice gets rewritten
Address state	ready or not	ready, serving, terminating
Read by kube-proxy	no	yes

For a quick manual check, the short kubectl get endpoints is still handy, which is why it keeps showing up in other guides. In scripts and new code, rely on EndpointSlice: Endpoints has nothing ahead of it but removal from the API.

Which Pods make it into the list

The controller adds a Pod to a Service's EndpointSlice when all of this holds:

1. The Pod is in the same namespace as the Service.
2. Every key-value pair from selector is present in the Pod's labels, values matching exactly.
3. The Pod hasn't finished: its phase isn't Succeeded or Failed. A Pod with STATUS Completed doesn't make the list.
4. The Pod already has an IP.

Making the list doesn't mean getting traffic yet. Traffic only goes to addresses with ready: true. A Pod is ready once all its containers pass their readinessProbe (or have none at all). A readinessProbe is the check Kubernetes uses to decide whether a container is ready to take traffic. While the probe is red, the address sits in the EndpointSlice with ready: false, and in Endpoints it moves into the notReadyAddresses field.

A Pod being deleted (deletionTimestamp set, STATUS Terminating) disappears from Endpoints immediately. In EndpointSlice it stays until deletion finishes, with ready: false and terminating: true. The serving field shows whether it's still passing its readinessProbe. Normally kube-proxy skips such addresses. Exception: if a Service is left with only Pods being deleted, traffic goes to whichever of them has serving: true, so requests don't get dropped during a rollout.

Break any of these conditions, and within a couple of seconds the address disappears from the list or turns not ready. kube-proxy stops routing traffic there.

What kubectl shows you

kubectl get endpoints api

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME   ENDPOINTS                         AGE
api    10.244.0.7:8080,10.244.0.9:8080   1s

The short output shows only ready addresses. Not-ready ones show up in describe:

kubectl describe endpoints api

...
Subsets:
  Addresses:          10.244.0.7,10.244.0.9
  NotReadyAddresses:  10.244.0.5
...

In EndpointSlice, each address has its own Conditions block and the Pod's name in TargetRef:

kubectl describe endpointslice -l kubernetes.io/service-name=api

...
Endpoints:
  - Addresses:  10.244.0.5
    Conditions:
      Ready:    false
    TargetRef:  Pod/notready
...
  - Addresses:  10.244.0.7
    Conditions:
      Ready:    true
    TargetRef:  Pod/ok-1
...

This is the main debugging tool. The Pod exists, but no traffic reaches it? If the address is in the list with Ready: false, check the Pod's readinessProbe. If the address isn't there at all, check the namespace, labels and status of the Pod. -o yaml shows the full object with every condition (serving, terminating):

kubectl get endpointslice -l kubernetes.io/service-name=api -o yaml

How a selector breaks

Three classic cases.

1. A typo in the selector. The Service looks for app: aip instead of api. The EndpointSlice is empty, and a request to the ClusterIP gets Connection refused:

wget: can't connect to remote host (10.96.190.198): Connection refused

Fixed by patching the Service:

kubectl patch svc api --type=merge -p '{"spec":{"selector":{"app":"api"}}}'

2. A Pod got relabeled. Someone ran kubectl label pod ... app=other --overwrite. The Pod is alive, but no longer matches the selector, and within a couple of seconds it drops out of the EndpointSlice. Fixed by restoring the label.

3. The Service is in one namespace, the Pod in another. A selector only works within the Service's own namespace. There's no field on a Service that extends it to another namespace.

Headless Service

If you set clusterIP: None on a Service, Kubernetes doesn't give it a virtual IP. A DNS query for that Service's name returns the addresses of ready Pods directly. The same controller still builds the EndpointSlice from the same selector, not-ready addresses included. The only difference is how the client gets the address: through DNS, not through kube-proxy's rules.

Service without a selector

You can create a Service without a selector. Then the controller doesn't write its addresses, you write them yourself. This is how an external address gets plugged into a cluster: a database on a separate server, a managed service, an old machine. Pods connect by the Service's name, and the IP lives in one object.

A Service without a selector:

apiVersion: v1
kind: Service
metadata:
  name: external-db
spec:
  ports:
    - port: 5432

The address comes from an EndpointSlice. It can have any name; the label kubernetes.io/service-name links it to the Service:

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: external-db-1
  labels:
    kubernetes.io/service-name: external-db
addressType: IPv4
ports:
  - port: 5432
endpoints:
  - addresses: ["10.0.0.42"]

The port in the Service and the port in the EndpointSlice are matched by name. Here neither has a name, so they match by default. If you name the port in one object, name it the same way in the other. The documentation also recommends tagging such slices with the label endpointslice.kubernetes.io/managed-by, naming whoever manages them (for example, your own team), so they don't get confused with slices owned by a controller.

Older guides do the same thing with a v1 Endpoints object named the same as the Service:

apiVersion: v1
kind: Endpoints
metadata:
  name: external-db
subsets:
  - addresses: [{ ip: 10.0.0.42 }]
    ports: [{ port: 5432 }]

This still works: kubectl prints the deprecation warning, and the endpointslicemirroring controller creates an EndpointSlice from that object, so traffic gets through. In new code, write the EndpointSlice directly.

Nobody watches the health of a manual address: there's no readinessProbe for an external server, so if it goes down, the address stays in the list.

This trick is a good illustration that the address list is a separate object. Usually a controller just keeps it for you.

Common mistakes

• The selector matches no Pod at all: a typo, an extra pair, or the Pod is in a different namespace. The EndpointSlice is empty, and the client gets Connection refused.
• The Pod is in the EndpointSlice but with ready: false: a red readinessProbe. The short kubectl get endpoints won't show that address at all.
• Only kubectl get endpointslice gets checked: that shows IPs with no conditions and no Pod names. Debugging needs describe or -o yaml.
• Looking for the EndpointSlice by the Service's name. A slice's name is random; look it up by the label kubernetes.io/service-name instead.
• A manual EndpointSlice without the label kubernetes.io/service-name, or with a port name that doesn't match the Service's port name: the addresses never get linked to the Service.

What you'll need for the tasks

• kubectl get endpointslice -l kubernetes.io/service-name=<name> and kubectl describe endpointslice -l ...: addresses, Conditions and TargetRef for each address.
• The conditions for a Pod to make the list: namespace, every selector pair, Pod not finished; only addresses with ready: true get traffic.
• kubectl get pods --show-labels and -n <namespace> to check labels.
• A Service without a selector plus an EndpointSlice with the label kubernetes.io/service-name, addressType, ports and endpoints.
• kubectl get nodes -o wide: the INTERNAL-IP column.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-endpoints-selector     # set up the starting state
./run.sh verify k8s-endpoints-selector    # check the current step