Labels and selectors

How cluster objects find each other by label, and how to filter on labels with -l.

Why this matters

A cluster has dozens of Pods, with random names like api-7d4b9c-x2kfp. A Service, a Deployment, and you yourself find the right Pods not by name, but by labels. If a label is wrong, a Service silently loses its Pods, and a Deployment stops counting them as its own. There's no error. This lesson covers how labels work, how to search by them, and how to change them.

What a label is

A label is a key-value pair in the metadata.labels field of any resource. Both the key and the value are strings.

metadata:
  name: web-1
  labels:
    app: web
    env: prod

A label isn't for description. Objects in the cluster find each other by it. A Service finds its Pods by label. A Deployment counts which replicas belong to it by label. A NetworkPolicy, a rule that restricts network traffic between Pods, also selects its Pods by label. A Pod with no labels will start and run, but no Service will find it.

There are limits on the key and the value. A key name is up to 63 characters: letters, digits, -, _, ., starting and ending with a letter or digit. The name can be preceded by a domain prefix with a slash, as in kubernetes.io/service-name. The prefixes kubernetes.io/ and k8s.io/ are reserved for system labels. A value is also up to 63 characters and has no spaces. The API server won't accept a label with a space:

kubectl label pod web-1 'owner=team a'

error: invalid label value: "owner=team a": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character ...

Labels and annotations

An annotation is a free-form note on an object (metadata.annotations). It looks like a label but serves a different purpose: it's data for people and tools. You can't filter by annotation, but its value can hold anything, even spaces or JSON. If you need the value for searching, put it in a label. If it's a note, put it in an annotation:

kubectl annotate pod web-1 note='checked by on-call'

pod/web-1 annotated

Trait	metadata.labels	metadata.annotations
Filter -l	works	doesn't work
Who reads it	Service, Deployment, NetworkPolicy	people and external tools
What to put there	a value to search by	a note
Value limits	up to 63 characters, no spaces	almost none

Filtering with -l

The flag -l (or --selector) filters resources by label. The simplest form checks for equality:

kubectl get pod -l app=web

The command shows only the Pods with the label app=web. You can combine conditions with a comma, which works as a logical AND:

kubectl get pod -l app=web,env=prod

Inequality is written with !=:

kubectl get pod -l env!=dev

There's also a more flexible set-based syntax:

kubectl get pod -l 'env in (prod, staging)'
kubectl get pod -l 'env notin (dev)'
kubectl get pod -l 'tier'
kubectl get pod -l '!tier'

The third command finds the Pods that have the label tier with any value. The fourth finds the Pods that don't have the label tier at all.

!= and notin have a trap. They also match Pods that don't have that key at all. Say the namespace has a Pod tool with only one label, app=tool. Then -l env!=dev will show it too: tool has no env, so by that logic it's also not dev. If you only want Pods that have env, add a check for existence: -l 'env,env!=dev'.

Put expressions with in, notin, and ! in single quotes. Without them the shell treats the parentheses and the exclamation mark as its own special characters:

kubectl get pod -l env in (prod)

bash: syntax error near unexpected token `('

The uppercase -L works differently. kubectl get pod -L app,env doesn't filter, it adds the values of these labels to the output as columns:

kubectl get pod -L app,env

NAME    READY   STATUS    RESTARTS   AGE   APP    ENV
db-1    1/1     Running   0          40s   db     prod
tool    1/1     Running   0          40s   tool
web-1   1/1     Running   0          40s   web    prod
web-2   1/1     Running   0          40s   web    staging

--show-labels shows all the labels at once:

kubectl get pod --show-labels

The kubectl label command

Set a label:

kubectl label pod web-1 tier=backend

pod/web-1 labeled

Change the value of an existing label. You need --overwrite, otherwise kubectl refuses:

kubectl label pod web-1 tier=frontend

error: 'tier' already has a value (backend), and --overwrite is false

kubectl label pod web-1 tier=frontend --overwrite

Remove a label: a - suffix after the key, with no value:

kubectl label pod web-1 tier-

pod/web-1 unlabeled

Apply to several Pods at once by selector:

kubectl label pod -l app=db owner=team-a
kubectl label pod --all owner-

-l app=db goes over every Pod that already has the label app=db. --all takes every Pod in the namespace with no conditions. If some Pods don't have the label being removed, kubectl prints label "owner" not found. for them and moves on.

How Service and Deployment select Pods

A Service looks for its Pods through the field spec.selector. This is a label selector on equality only: every pair in it has to match.

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080

Label keys are case-sensitive. app and App are different keys. If the selector says App: web while the Pods carry app: web, the Service won't find a single Pod. There's no error, the list of addresses behind the Service just stays empty.

That list is stored in EndpointSlice objects. Kubernetes puts the label kubernetes.io/service-name on them with the Service's name, so you search for them with a selector too:

kubectl get endpointslice -l kubernetes.io/service-name=web

NAME        ADDRESSTYPE   PORTS   ENDPOINTS                 AGE
web-8x2kq   IPv4          8080    10.244.0.7,10.244.0.8     2m

If the selector didn't match any Pod, the ENDPOINTS column shows <unset>. The older command kubectl get endpoints web shows the same thing, but the v1 Endpoints object is deprecated since Kubernetes 1.33, and kubectl prints a warning:

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice

If you remove or change a running Pod's label that's part of a Service selector, the Pod drops out of the Service. From the outside it looks like "the Pod is Running, but there's no traffic."

A Deployment keeps its Pods through spec.selector.matchLabels. Here you can also use set-based expressions (matchExpressions with In, NotIn, Exists, DoesNotExist). The labels in the Pod template must match the selector, and the selector itself can't be changed after the Deployment is created.

If you remove or change a Pod's label that's part of matchLabels, the Deployment stops seeing it and brings up a new Pod to replace it. The old Pod stays hanging in the namespace with no controller managing it: it loses its owner reference (metadata.ownerReferences). This is useful during incident investigation: the Pod is alive and you can inspect it, but it no longer affects traffic or the replica count.

Labels that aren't part of the Deployment's selector can be changed on live Pods as much as you like, the Deployment doesn't react to that. The flip side: it won't restore them either. The Pod template only affects new Pods.

Common mistakes

• A Service selector with the wrong case or a typo in the key. No error, the EndpointSlice is just empty.
• A bulk kubectl label pod -l ... --overwrite with too wide a selector. Labels change on Pods you never meant to touch. Before editing, run kubectl get pod -l <the same selector> and see who's caught.
• -l env!=dev or notin relying only on Pods that have the label env. Pods with no env at all will show up too.
• A set-based selector with no quotes: syntax error near unexpected token from the shell, the command never reaches kubectl.
• A label instead of an annotation for a note with spaces: invalid label value.
• A repeated kubectl label with no --overwrite: already has a value.

What you'll need for the tasks

• kubectl get pod --show-labels and kubectl get pod -L <keys>, to see the labels.
• kubectl get pod -l <selector>: equality, !=, in, notin, checking for key existence.
• kubectl label pod: set, --overwrite, remove with key-, by selector -l.
• kubectl annotate pod <name> <key>=<value>.
• kubectl get endpointslice -l kubernetes.io/service-name=<Service>, to see which Pods are behind a Service.
• kubectl get svc <name> -o jsonpath='{.spec.selector}', to read a Service's selector.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your computer, instructions are in its README.md.

Homework

Practice the theory on your own cluster: 3 steps with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-labels-selectors     # set up the starting state
./run.sh verify k8s-labels-selectors    # check the current step

In short

• A label is a key-value pair in metadata.labels, objects find each other by it.
• An annotation looks like a label, but you can't filter by it, and its value can hold anything.
• -l filters by labels, -L adds them to the output as columns.
• Put set-based selectors (in, notin, !tier) in quotes. != and notin match objects with no such key too.
• kubectl label sets, changes (--overwrite), and removes (key-) labels, including in a group with -l or --all.
• Keys are case-sensitive: a selector App won't find a Pod with app, and there's no error, the EndpointSlice just stays empty.
• Change a label that's part of a Deployment selector, and the Pod is orphaned and the Deployment brings up a replacement.