Service ClusterIP: an internal address for Pods

How a group of Pods gets one stable address and DNS name inside the cluster, and how to check that a Service really leads to live Pods.

Why you need a Service

Pod: the smallest unit that Kubernetes runs. Inside a Pod, one or more containers run. Each Pod gets its own IP address when it starts.

This IP changes on any Pod recreation. A node went down, a rollout happened (replacing a Pod with a new version), someone changed the number of replicas. The IP will be different. You can't hardcode a Pod's IP in a neighboring application's code: a minute later it's already gone.

Service: a Kubernetes object that gives a group of Pods a stable address. This address doesn't depend on which exact Pods are alive right now.

What a Service does

A Service does four things.

• Gets a stable virtual IP inside the cluster (field spec.clusterIP).
• Gets a DNS name <svc-name>.<namespace>.svc.cluster.local. Namespace: a scope, a logical partition of the cluster for grouping objects.
• Finds its Pods by labels through the selector field.
• Distributes requests across those Pods. In the default kube-proxy mode (iptables), a backend is picked at random for each connection.

A Service doesn't check Pod health itself. It watches the Pod's readinessProbe. readinessProbe: a check that Kubernetes repeats to find out whether a container is ready to take traffic. A Pod without a readinessProbe is considered ready as soon as its containers start. A Pod with a readinessProbe only joins the rotation after its first successful check.

The ClusterIP manifest

The examples in this lesson use namespace demo and Deployment web with two replicas. The container listens on port 8080 and responds with the string web <Pod name>.

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: demo
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP

type: ClusterIP

ClusterIP: the Service's internal address. It's reachable only from Pods and nodes in the cluster. It's not visible from outside the cluster.

ClusterIP is the default, you can skip the type field. The other types (NodePort, LoadBalancer, ExternalName) solve different problems and are covered in separate lessons.

The address comes from a separate range set aside for Services, chosen when the cluster is created. In kind it's 10.96.0.0/16, other distributions use their own range. It doesn't overlap with Pod addresses.

selector

selector: a set of key: value pairs. The Service looks in its own namespace for every Pod that has these labels, and treats them as its own.

selector:
  app: web

If no Pod matches, the Service exists, but its list of addresses is empty. Requests to it come back with Connection refused. A typo in selector breaks the connection silently: the Service still looks fine.

ports

ports: the Service's list of ports. Usually just one. A port has three keys.

• port: the Service's own port. Clients use it: web:80.
• targetPort: the container's port in the Pod, where the traffic goes. In the example it's 8080.
• protocol: TCP by default, UDP or SCTP are also options.

Telling port and targetPort apart matters. The client sees port. kube-proxy forwards the request to targetPort. kube-proxy: a component on every node that sets up the network rules for a Service. If the Pod listens on 8080, but the manifest has targetPort: 80, the Pod's addresses are in the Service's list, but every request gets Connection refused.

What kubectl shows

kubectl: the command-line tool for working with the cluster.

kubectl get svc web -n demo

NAME   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
web    ClusterIP   10.96.38.121   <none>        80/TCP    18s

kubectl get endpoints web -n demo

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME   ENDPOINTS                         AGE
web    10.244.0.5:8080,10.244.0.6:8080   18s

The warning in the first line is normal. Starting with version 1.33, the Endpoints object is deprecated, replaced by EndpointSlice. The command keeps working and stays the shortest one for a quick check. The same list through the new API:

kubectl get endpointslice -n demo -l kubernetes.io/service-name=web

NAME        ADDRESSTYPE   PORTS   ENDPOINTS               AGE
web-dhf8n   IPv4          8080    10.244.0.5,10.244.0.6   18s

Kubernetes creates Endpoints and EndpointSlice itself next to every Service that has a selector. Inside is the list of real Pod addresses that currently match the selector, and the targetPort. When a Pod stops passing its readinessProbe or starts terminating, its address is marked not ready right away, and traffic stops going to it. In EndpointSlice, such an address shows up with ready=false, in Endpoints it's just not there.

Cross-check the pair like this:

kubectl get pod -n demo -l app=web -o wide

NAME                  READY   STATUS    RESTARTS   AGE   IP           ...
web-79cbd46fb-4zvv5   1/1     Running   0          18s   10.244.0.5   ...
web-79cbd46fb-ksr6b   1/1     Running   0          18s   10.244.0.6   ...

The Pod IPs from this output should match the addresses in the EndpointSlice. An empty list when Pods exist and are Ready means the selector didn't find any Pods.

DNS inside the cluster

CoreDNS: the cluster's DNS server. It runs in namespace kube-system and translates Service names into their ClusterIP. From a Pod in namespace demo (the busybox image has wget instead of curl):

# Short name: only from a Pod in the same namespace
wget -qO- http://web/

# Name with namespace: from any namespace
wget -qO- http://web.demo/

# Full name (FQDN)
wget -qO- http://web.demo.svc.cluster.local/

All three variants work thanks to the search line in /etc/resolv.conf. kubelet puts this file into every Pod. kubelet: the Kubernetes agent on a node that runs Pods. For a Pod from demo, the file looks like this:

search demo.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

The short web gets the first suffix appended and turns into web.demo.svc.cluster.local. From a Pod in namespace default, the first suffix would be default.svc.cluster.local, which is why you write web.demo to reach it.

The fast imperative command

If a Deployment already exists, you don't need to write YAML by hand. Deployment: an object that keeps a set number of identical Pods running.

kubectl expose deploy web -n demo --port=80 --target-port=8080 --name=web

The command creates a Service with the selector taken from the Deployment's spec.selector and the right ports. --type=ClusterIP by default. If you don't give --target-port, kubectl takes the value of --port, and that may not match the container's port.

Common mistakes

• Pod labels don't match the Service's selector. The list of addresses is empty. Shown by kubectl get pod --show-labels and kubectl describe svc web (the Selector and Endpoints lines).
• targetPort doesn't match the port the container listens on. The addresses are in the list, but requests get Connection refused. Cross-check the Pod's containerPort, the PORTS column in EndpointSlice, and the port the application actually listens on.
• Pod in a different namespace. A selector doesn't cross a namespace boundary: a Service in default with app: web won't find a Pod from demo. From another namespace, you reach a Service by its namespaced name (web.demo).
• Short name from a different namespace. http://web/ from a Pod in default looks for web.default.svc.cluster.local, not the Service in demo.
• A Pod without a readinessProbe gets traffic before the application has started. The first requests after a rollout fail with Connection refused.

What you'll need for the tasks

The tasks live in the labs/ folder: each lab brings up its own kind cluster on your computer, the instructions are in its README.md. These commands will come in handy:

• kubectl get svc, kubectl describe svc <name>;
• kubectl get endpointslice -l kubernetes.io/service-name=<name>;
• kubectl get pod --show-labels, kubectl get pod -o wide;
• kubectl describe deploy <name> (the template's labels and container port);
• kubectl exec <pod> -- wget -qO- -T 3 http://<name>/ for a request from inside the cluster;
• kubectl expose, kubectl edit svc, kubectl apply -f.

Homework

Practice the theory on your own cluster: 4 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-service-clusterip     # set up the starting state
./run.sh verify k8s-service-clusterip    # check the current step

In short

• A Service gives a group of Pods a stable ClusterIP and DNS name that don't depend on any one Pod's lifetime.
• Pods are found through selector by labels, and only inside their own namespace.
• port is the Service's port for clients, targetPort is the container's port. Mixing them up breaks traffic silently.
• The list of real addresses lives in Endpoints (deprecated) and EndpointSlice. An empty list means the selector found no Pods, or none of them are Ready.
• kubectl expose deploy creates a ClusterIP Service without manual YAML.