The Service isn't responding: selector mismatch

The Deployment is rolled out, the Pods are Running and Ready, the Service is created. You hit the Service and the client gets Connection refused. The app logs are empty: the requests simply never arrive.

A common cause behind this picture: the Service selector doesn't match the Pod labels. In this lesson you'll learn to spot that mismatch in two commands and fix it without taking the app down.

How a Service finds Pods

A Service is a Kubernetes object that gives a group of Pods a stable address (ClusterIP) and a DNS name. It doesn't know the Pods themselves. It only knows the selection rule: the spec.selector field, a set of key=value pairs.

The EndpointSlice controller looks at which Pods have labels that match this selector, and writes their IPs into an EndpointSlice object. It only looks in the namespace the Service lives in. Then kube-proxy, a component on every node, reads the EndpointSlice and sets up network rules: packets to the ClusterIP go to the IPs of specific Pods.

If the selector doesn't match, the EndpointSlice has no addresses and the packet has nowhere to go. On a cluster with kube-proxy in iptables mode (as in this lesson's lab), the client gets an immediate refusal:

wget: can't connect to remote host (10.96.215.189): Connection refused

The matching rule

A Pod makes it into a Service if it has every pair from the selector with the same value. The comparison is exact: no substrings, no regular expressions, and a Service has no operators like In.

Take the selector {app: webapp, tier: frontend}:

• a Pod with labels {app: webapp, tier: frontend, version: v1} fits, extra labels don't get in the way;
• a Pod with {app: webapp} doesn't fit, there's no tier key;
• a Pod with {app: webapp, tier: front-end} doesn't fit, the value differs by one hyphen.

The last case is the trickiest. The app key matches, your eye catches that and stops reading.

The key indicator: EndpointSlice

When a Service isn't responding, the first thing to check is its EndpointSlice. Every slice has a kubernetes.io/service-name label with the Service's name, that's what you search by:

kubectl get endpointslice -l kubernetes.io/service-name=webapp

If the selector didn't match anyone, the slice exists but has no addresses in it:

NAME           ADDRESSTYPE   PORTS     ENDPOINTS   AGE
webapp-2pz7m   IPv4          <unset>   <unset>     16s

The ENDPOINTS column shows all the addresses found, including addresses of Pods that haven't passed the readiness check yet. Traffic doesn't go to those Pods. To see which addresses are ready, add a column with the conditions.ready field:

kubectl get endpointslice -l kubernetes.io/service-name=webapp \
  -o custom-columns='IP:.endpoints[*].addresses[0],READY:.endpoints[*].conditions.ready'

IP                      READY
10.244.0.5,10.244.0.6   true,true

That gives you three outcomes. No addresses: look for a mismatch between the selector and the labels (or there are no matching Pods at all). Addresses present but false: the Pods were found, the problem is in the readiness check. Addresses present and true: the selector is fine, dig into ports and network policies.

This same information used to be checked through kubectl get endpoints webapp. The v1 Endpoints object has been deprecated since Kubernetes 1.33, and kubectl warns about it:

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME     ENDPOINTS   AGE
webapp   <none>      16s

The command still works, but its ENDPOINTS column only shows ready addresses. If no Pod is Ready, the column is empty, and that case is easy to mistake for a selector mismatch.

kubectl describe svc webapp also shows ready addresses, in the Endpoints: line. It's empty when there's a mismatch.

How to compare the selector and the labels

The Service's selector:

kubectl get svc webapp -o jsonpath='{.spec.selector}'

{"app":"webapp","tier":"front-end"}

The Pods' labels:

kubectl get pods --show-labels

NAME                      READY   STATUS    RESTARTS   AGE   LABELS
webapp-5b7fd9946b-ncmnh   1/1     Running   0          17s   app=webapp,pod-template-hash=5b7fd9946b,tier=frontend
webapp-5b7fd9946b-tg9lc   1/1     Running   0          17s   app=webapp,pod-template-hash=5b7fd9946b,tier=frontend

Now compare the pairs one by one. app=webapp is on both sides. But tier=front-end on the Service and tier=frontend on the Pod don't match. The pod-template-hash label is added by the Deployment itself, it doesn't affect the match as long as it isn't in the selector.

There's also a quick way: select Pods with the same selector the Service has.

kubectl get pods -l app=webapp,tier=front-end

The answer No resources found confirms that no Pod fits this selector.

Cross-check: reach the Pod directly

Make sure the app itself is alive. Find one Pod's IP and hit it directly from a throwaway Pod:

kubectl get pod -l app=webapp -o jsonpath='{.items[0].status.podIP}'

kubectl run probe --image=busybox:1.36.1 --rm -it --restart=Never -- \
  wget -qO- -T 5 http://<POD_IP>:8080

The port here is containerPort, the one the container listens on, not the Service's port. Without :8080, wget would go to port 80 and get Connection refused, even though the app is fine.

If the Pod's IP answers but the Service doesn't, the cause is the selector or the Service's ports. The app has nothing to do with it.

In a namespace under Pod Security restricted, this kubectl run won't go through: admission will answer Forbidden, because the probe Pod has no securityContext. The error text lists the fields you need to set.

Selector, port, or network policy

From the outside these defects all look the same: "the service doesn't work". The EndpointSlice is what tells them apart.

What the client sees	Addresses in the EndpointSlice	Cause
Connection refused	none	selector doesn't match the labels
Connection refused	present, ready=false	Pods haven't passed the readiness check
Connection refused	present, ready=true	targetPort doesn't match the port the container listens on
timeout	present, ready=true	a NetworkPolicy is blocking the traffic

The client can't tell the first three rows apart. That's why you check the EndpointSlice first: it tells you right away which side to dig into. On a cluster with a different dataplane, the "What the client sees" column may look different. The "Addresses in the EndpointSlice" column doesn't depend on that.

How it's fixed

The mismatch gets removed from one of the two sides, and the sides aren't equally good options.

Fixing the Service's selector is one command, and the EndpointSlice fills in within seconds:

kubectl patch svc webapp -p '{"spec":{"selector":{"tier":"frontend"}}}'

This command only changes the tier key, the rest of the selector's keys stay as they were. If an extra key needs to go away entirely, pass it null in a merge patch:

kubectl patch svc webapp --type=merge -p '{"spec":{"selector":{"<key>":null}}}'

kubectl patch here is an emergency measure. In production, a Service usually comes from Helm, Kustomize, or Argo CD, and a manual edit drifts from the manifest in Git. Argo CD with self-heal enabled will revert the object to the state from Git (docs). The permanent fix goes into the manifest or the chart's values and ships through the normal pipeline.

Fixing Pod labels through the Deployment is harder. An existing Deployment's spec.selector field can't be changed. The selector ties the Deployment to its ReplicaSet and Pods. A new selector wouldn't see the old ReplicaSets, and they'd be left orphaned (docs). The API responds like this:

The Deployment "webapp" is invalid:
* spec.selector: Invalid value: {"matchLabels":{"app":"webapp","tier":"front-end"}}: field is immutable

Labels in the Pod template (spec.template.metadata.labels) can be changed, but they must still cover the Deployment's selector. Adding a new label works. Changing a value that's part of the selector doesn't:

The Deployment "webapp" is invalid: spec.template.metadata.labels: Invalid value: {"app":"webapp","tier":"front-end"}: `selector` does not match template `labels`

So to change the Pod labels that are in the selector, the Deployment has to be recreated. Without downtime, that's done like this: bring up a new Deployment alongside it with the right labels, wait for it to be Ready, switch the Service over, and only then delete the old one.

That gives a practical rule: on a mismatch, you usually fix the Service. Exception: the Pod labels are wrong in meaning, say app: web instead of the project's accepted app: webapp. Then you fix the labels in the Deployment, even if it means recreating it.

It's best not to keep the app version in a Service selector. Otherwise every release needs a Service edit, and while Pods of two versions live side by side during the rollout, only one of them gets traffic. It's a different story when traffic is deliberately switched between two Deployments, as in the no-downtime approach above. There, the version pair in the Service selector is exactly what does the switching.

Common mistakes

Rewriting live Pods' labels. The command kubectl label pods ... --overwrite looks like a quick fix. If the rewritten label is part of the Deployment's selector, the Pods fall out of their ReplicaSet: its selector no longer sees them. The ReplicaSet immediately creates a replacement, and now you have twice as many Pods, half of them orphaned. The Service only works through those orphaned Pods, and the Deployment's new replicas never make it in.

If the label isn't part of the Deployment's selector, the Pods stay in the ReplicaSet and the Service works for now. But the Pod template hasn't changed: any new Pod after a restart or a rollout comes up with the old label and doesn't make it into the Service.

Checking Pods in the wrong namespace. kubectl get pods --show-labels without -n shows the namespace from your current context. A Service only looks for Pods in its own namespace, so compare labels in that same namespace the Service lives in.

Assuming an empty EndpointSlice is always a bug. A Service can have no selector at all and have its EndpointSlice filled in by hand, for example to point traffic at a database outside the cluster. Kubernetes doesn't touch that kind of slice on its own.

How to avoid it

Pull labels and selectors from one place, instead of writing them by hand in two files.

A Helm template from helm create already has a <chart>.selectorLabels helper. It gets used in both the Deployment's selector and the Service's selector.

In Kustomize there's a labels field with includeSelectors: true for this:

labels:
- pairs:
    app: webapp
  includeSelectors: true

The old commonLabels field does the same thing, but it's deprecated. kubectl prints Warning: 'commonLabels' is deprecated. Please use 'labels' instead.

After rolling out a new Service, it's worth checking that it has ready addresses. Right after apply the Pods aren't Ready yet, so in CI you first wait for the rollout, then print the ready addresses. Empty output means the rollout is broken. kubectl get itself returns exit code 0 on empty output, so the CI step fails on grep -q . instead:

kubectl rollout status deployment/webapp --timeout=120s
kubectl get endpointslice -l kubernetes.io/service-name=webapp \
  -o jsonpath='{range .items[*].endpoints[?(@.conditions.ready==true)]}{.addresses[0]}{"\n"}{end}' \
  | grep -q . || { echo "Service webapp has no ready addresses"; exit 1; }

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-service-selector-mismatch     # set up the starting state
./run.sh verify k8s-service-selector-mismatch    # check the current step

In short

• A Service finds Pods only by labels: it needs every selector pair, values matching exactly.
• Check first with kubectl get endpointslice -l kubernetes.io/service-name=<svc>: no addresses means a selector/label mismatch.
• Addresses are there but you still get Connection refused: check ready and targetPort.
• A Deployment's selector can't be changed, so on a mismatch you usually fix the Service.
• One source for labels in Helm or Kustomize removes this bug entirely.

What you'll need for the task

• kubectl config current-context: make sure you're working in the lab cluster.
• kubectl get endpointslice -l kubernetes.io/service-name=<svc>.
• kubectl get svc <svc> -o jsonpath='{.spec.selector}' and kubectl get pods --show-labels.
• kubectl patch svc <svc> -p '{"spec":{"selector":{...}}}' and the variant with --type=merge and null to remove a key.