Pod Running, but the client sees a 502

This incident feels like a bad joke. The Pod is Running, zero restarts, the logs are clean, the app inside the container responds. And users get a 502.

This picture almost always means a break between the Service and the Pod. The ingress controller or the proxy in front of the app has nothing to do with it: it's honestly passing along whatever the upstream returned.

Three different ports in one chain

A Service looks simple until someone confuses the ports that are named almost the same. It helps to think of them as a pipeline:

client → svc.clusterIP:port → pod:targetPort → listener (the process in the container)

• port. The Service's own port inside the cluster (ClusterIP:port). Clients from inside the cluster talk to this.
• targetPort. The port on the Pod that the Service sends traffic to. This is a number or the container port's name.
• containerPort. The port declaration in the container's spec. It doesn't affect traffic: the Service will deliver packets to any port a process listens on, even one that isn't declared (Pod API, the ports field).
• listener. The port the process inside the container is actually listening on.

Only targetPort and the listener are required to match. containerPort can lie, and Kubernetes won't notice.

There's no nodePort in this chain: with type=NodePort it would add an external hop, nodeIP:nodePort, before svc.port. With a ClusterIP Service, nothing is visible from outside the node.

A Service doesn't check whether targetPort is open inside the Pod. It just sends traffic there. If the app listens on :8080, and the Service has targetPort: 80, the connection hits a closed port. A request to the ClusterIP returns Connection refused, and the same path through nginx in front of the Service hands the client a 502. The nginx log shows exactly what happened:

connect() failed (111: Connection refused) while connecting to upstream, ... upstream: "http://10.96.99.182:80/"

Without a readinessProbe on the same port, a Pod stays Ready with a closed listener: it lands in the EndpointSlice and gets traffic. The rollout goes green, and the client sees a 502.

Check a ClusterIP Service from inside the cluster, from another Pod, not from outside:

kubectl exec deploy/gateway -- wget -qO- -T 5 http://webapp.demo.svc:80/

The symptom: 502 or Connection refused

In the examples below, kube-proxy runs in iptables mode, with nginx in front of the Service.

What you see	What it means
Connection refused at ClusterIP:port	the EndpointSlice has no ready addresses, OR targetPort hits a closed port
502 through nginx or an ingress controller	the same thing, just seen through the proxy's eyes: the connection never reached the Pod

Both causes look the same to the client. The EndpointSlice tells them apart: an empty list means a selector or readiness problem, a list with addresses means a port problem.

Step-by-step diagnosis

Don't guess. Follow the chain.

Step 1. Does the Service see the Pod?

kubectl get endpointslice -l kubernetes.io/service-name=webapp

If the ENDPOINTS column shows <unset>, the Service's selector doesn't match the Pod's labels. That's the previous lesson's topic.

If there's an IP in ENDPOINTS, the Pods are hooked up, and the PORTS column shows where the traffic will go:

NAME           ADDRESSTYPE   PORTS   ENDPOINTS               AGE
webapp-84stw   IPv4          8080    10.244.0.5,10.244.0.6   105s

This used to be checked with kubectl get endpoints webapp. The v1 Endpoints object was deprecated starting with Kubernetes 1.33, and kubectl prints a warning about it, but the command still works.

Step 2. Which port is recorded in the EndpointSlice?

kubectl get endpointslice -l kubernetes.io/service-name=webapp \
  -o jsonpath='{.items[0].ports[0].port}{"\n"}'

This is the Service's targetPort, translated into a number. If targetPort is given as a name, the controller finds the port with that name in the Pod and writes its number here. kube-proxy doesn't compute anything, it just reads the ready-made EndpointSlice.

Step 3. Is the Pod actually listening on this port?

POD=$(kubectl get pod -l app=webapp -o name | head -1)
kubectl exec "$POD" -- netstat -tln

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 :::9090                 :::*                    LISTEN

busybox and alpine have netstat, but usually no ss: kubectl will answer executable file not found in $PATH. If you have neither, knock on the port directly:

kubectl exec "$POD" -- wget -qO- -T 3 http://localhost:8080/

wget: can't connect to remote host (127.0.0.1): Connection refused

The listener is on one port, and the Service sends traffic to another. There's your break.

Step 4. Compare the specs

kubectl get svc webapp \
  -o jsonpath='{"port="}{.spec.ports[0].port}{" targetPort="}{.spec.ports[0].targetPort}{"\n"}'

kubectl get deploy webapp \
  -o jsonpath='{.spec.template.spec.containers[0].ports[*].containerPort}{"\n"}'

What should match:

• the Service's targetPort equals the port the app is listening on;
• the Deployment's containerPort is best set to the same port. For a Service to work by number, it isn't required. It's required when the port is referenced by name: in targetPort or in a probe.

Don't trust containerPort as a source of truth. If in step 3 the listener differs from both targetPort and containerPort, both are lying.

The break shows up fastest in a single output:

kubectl describe svc webapp

Selector:                 app=webapp
...
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
Endpoints:                10.244.0.5:8080,10.244.0.6:8080

describe prints TargetPort, Endpoints, and Selector side by side. It doesn't show the listener itself, you only see that from step 3.

Where the mismatch comes from

• The Service was created with kubectl expose deploy webapp --port=80: targetPort defaults to port, that is, 80. The container listens on 8080, and the Service points into nothing. Without --port, expose takes the port from the Deployment's containerPort.
• A manifest was copied from nginx (port 80) for nginx-unprivileged (port 8080): containerPort and the listener shifted, targetPort stayed put.
• Someone changed the image or the port variable in the Deployment, the listener moved, and nobody touched the Service manifest.

How to avoid stepping on this again

Keep the app's port in one place in the manifest. Kubernetes itself has a mechanism for this: a container port can get a name, and the Service and the probe reference the name, not the number (Service, named ports, probe with a named port):

# Deployment, the container
ports:
  - name: web
    containerPort: 3000
readinessProbe:
  httpGet:
    path: /healthz
    port: web
---
# Service
ports:
  - port: 80
    targetPort: web

The number 3000 is now recorded once. If the listener moves to a different port, you fix containerPort, and the Service and probe follow. A port name can be at most 15 characters: lowercase letters, digits and hyphens only.

A readinessProbe on the same port closes the other half of the problem too. If a new version listens on the wrong port, its Pods won't become Ready. The rollout gets stuck at the first new Pod, and the old ones keep serving traffic:

NAME                       READY   STATUS    RESTARTS   AGE
webapp-5cc569787d-jmt9k    0/1     Running   0          38m
webapp-66759cd48f-8zsfg    1/1     Running   0          39m
webapp-66759cd48f-jw7fl    1/1     Running   0          39m

kubectl rollout status doesn't finish, and CI sees the failed rollout before the client sees a 502.

Common mistakes

Fixing the app instead of the Service. The Pod is Running, logs are clean, and someone restarts the Pod and reads the code. Until you've looked at the EndpointSlice, it's guesswork.

Trusting containerPort. It's written by hand, and it can disagree with the real listener. Check the process in the container (step 3).

Changing port instead of targetPort. port is what the Service itself listens on, and clients and the proxy are configured for it. Shift it, and you break them.

Checking ClusterIP from your laptop. ClusterIP is reachable only from inside the cluster. A request from the host will never reach it, no matter the ports, and you'll be fixing something that isn't broken. Check from a Pod, through kubectl exec.

Homework

Practice the theory on your own cluster: 2 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-port-mismatch-502     # set up the starting state
./run.sh verify k8s-port-mismatch-502    # check the current step

In short

• Pod Running plus a 502 for the client: look at the Service, not the app.
• kubectl get endpointslice -l kubernetes.io/service-name=<svc> separates a selector problem from a port problem.
• The port in the EndpointSlice is targetPort as a number; compare it with the real listener inside the container.
• kubectl describe svc shows TargetPort, Endpoints, and Selector together.
• A named port with a readinessProbe on it turns a silent break into a stuck rollout.

What you'll need for the tasks

• kubectl config current-context: make sure you're working in the lab's cluster.
• kubectl get endpointslice -l kubernetes.io/service-name=<svc> and kubectl describe svc <svc>.
• kubectl exec <pod> -- netstat -tln and kubectl exec <pod> -- wget -qO- -T 3 http://localhost:<port>/.
• kubectl logs deploy/<name>, to see what the proxy writes.
• kubectl edit or kubectl patch for the Service and the Deployment; kubectl rollout status deployment/<name>.