Cluster architecture

Kubernetes isn't one program. It's seven or eight processes spread across nodes, talking to each other over the network. They follow one rule: everyone goes through kube-apiserver, and no component talks to another directly.

Why does this matter early on? When a Pod sits in Pending or kubectl answers with connection refused, the first thing you need to know is which component is responsible for that step. Without a map of the cluster, you're left guessing.

Concepts

• Node — a machine in the cluster: virtual, physical, or, like in this course's labs, a Docker container. Pods run on nodes.
• control plane — the components that manage the cluster: kube-apiserver, etcd, kube-scheduler, kube-controller-manager. They usually run on separate nodes.
• worker node — a node where application Pods run. It runs kubelet, kube-proxy, and the container runtime.
• watch — a subscription to changes. A component asks apiserver once, "tell me about new Pods," and from then on gets events instead of polling it in a loop.
• static Pod — a Pod that kubelet starts on its own from a YAML file in a folder on the node, without a command from apiserver. This is often how the control plane itself gets started: kube-apiserver can't create itself through itself.

Map

kubectl: a client to the API

kubectl reads ~/.kube/config (or the file from the KUBECONFIG variable), takes the cluster address and certificate from it, and makes a plain HTTPS request to kube-apiserver. There's no magic inside: you can send the same request with curl and get the same JSON back. apiserver does all the work; kubectl just prints the response nicely.

You can see which requests go out with the -v=6 flag:

kubectl get nodes -v=6

I1009 10:41:01.326043   42560 envvar.go:195] "Feature gate default state" feature="InformerResourceVersion" enabled=true
...
I1009 10:41:01.444211   42560 round_trippers.go:632] "Response" verb="GET" url="https://127.0.0.1:56793/api/v1/nodes?limit=500" status="200 OK" milliseconds=110
NAME                                 STATUS   ROLES           AGE   VERSION
...

There are many boilerplate lines at the start; you need just one: "Response" verb="GET" url=.... It has the method, the full request URL, and the response code.

kubectl does read the manifest itself: it parses the YAML and complains about broken indentation or a missing apiVersion or kind. But apiserver checks the fields: since version 1.27, server-side field validation has been stable, and kubectl asks the server by default to reject unknown fields. That's why the server rejects contaienrs instead of containers.

Who's responsible for what

kube-apiserver

A REST interface on top of etcd. It validates requests, authenticates clients, and enforces RBAC. Every other component talks to it, and nowhere else. apiserver itself stores no data, so it's easy to run several copies: production usually runs 3 instances behind a load balancer.

etcd

A distributed key-value store built on the Raft algorithm. It holds all cluster state: objects, Secrets, ConfigMaps, RBAC rules. Production runs etcd on 3 or 5 nodes, so a quorum survives losing one. One node is enough for a lab cluster.

kube-scheduler

It watches for Pods that don't have a node yet and decides where to place them. It accounts for requested resources, taints, affinity, and priorities. Once it picks a node, the scheduler writes that node's name into the Pod through apiserver (the bind operation). It doesn't start containers: from there, the kubelet on the chosen node picks up the Pod.

kube-controller-manager

One binary, with dozens of controllers inside: Deployment, ReplicaSet, Job, Node, EndpointSlice. Each one compares the desired state (5 replicas) with the real state (3 replicas) and uses apiserver to bring the real state in line. This is the reconcile loop, and everything Kubernetes does on its own rests on it.

cloud-controller-manager

Runs in cloud clusters (EKS, GKE, AKS) and anywhere a cloud provider is configured. It turns Kubernetes objects into cloud resources: a Service of type LoadBalancer becomes a cloud load balancer, and routes to Pods get written into the cloud's network. Bare-metal servers with no cloud don't run this process at all.

kubelet

The agent on every node. It gets Pods assigned to its node through watch, asks the container runtime to start the containers, mounts volumes, and sends the Pod status and the node's heartbeat to apiserver. If the heartbeat doesn't arrive for 50 seconds, the node gets the status Unknown. The threshold is set by the --node-monitor-grace-period flag on kube-controller-manager.

kubelet runs on the node as a regular process (a systemd service), not as a Pod: it's the one that starts Pods on the node in the first place.

kube-proxy

Also runs on every node. It watches Service and EndpointSlice through apiserver and writes rules into the Linux kernel. These rules are what turn a packet sent to a ClusterIP into the address of one of the Pods (DNAT).

There are three modes on Linux:

• iptables — the default mode;
• nftables — stable since 1.33, rules update faster and don't slow down with thousands of Services; it will become the default mode in a future version;
• ipvs — deprecated as of 1.35, nftables is recommended instead.

Which mode a specific cluster uses is set by kube-proxy's own configuration, not by the Kubernetes version. That's why you always check the mode on the spot.

Some clusters skip kube-proxy entirely: Cilium and Calico in eBPF mode handle Service traffic themselves.

container runtime

Starts containers. kubelet talks to it over CRI (a gRPC protocol), the runtime pulls the image from a registry and starts the process through runc. containerd is the standard today, CRI-O is the alternative. Kubernetes 1.24 removed dockershim (the layer that talked to Docker), but images built with Docker still work as before: they're just ordinary OCI images.

CNI plugin

Hands each Pod an IP address and sets up routes between nodes. The container runtime calls it when it creates a Pod's network environment: since version 1.24, kubelet no longer manages Pod networking itself. There are many implementations: Calico, Cilium, Flannel, AWS VPC CNI. Without a CNI plugin, Pods get no network, and the node never becomes Ready.

CNI and kube-proxy work at different layers: CNI gives Pods their network, and kube-proxy builds Service on top of it.

Everything goes through apiserver

On the diagram, every arrow leads to kube-apiserver. kubelet doesn't talk to etcd, the scheduler doesn't talk to kubelet, controller-manager doesn't read etcd. Every component reads and writes objects through apiserver and subscribes to their changes through watch.

One exception goes the other way: apiserver connects to kubelet itself when it needs container logs, kubectl exec, or port-forward. That's why kubectl logs fails when the node is unreachable, even while apiserver is alive.

Two things follow from this. apiserver is the busiest part of the control plane, which is why production runs it as several copies. And it's the single biggest point of failure: if apiserver goes down, you can't manage the cluster. Pods on the nodes keep running, but nothing restarts a crashed one or schedules a new one.

A cloud cluster vs. your own

In a cloud cluster (EKS, GKE, AKS), the control plane is hidden entirely: you pay the cloud and see only the worker nodes and the kube-apiserver address. On your own servers, you bring up and maintain the control plane yourself, and you usually don't need cloud-controller-manager there.

Common mistakes

• Looking for kubelet among the Pods. kubectl get pods -A shows kube-apiserver, etcd, kube-scheduler, and kube-proxy, but not kubelet: it's a process on the node. You check its logs on the node itself (journalctl -u kubelet).
• Assuming the scheduler starts Pods. The scheduler only picks a node and writes it into the Pod. If a Pod is assigned to a node but won't start, check kubelet and the runtime on that node, not the scheduler.
• Assuming kube-proxy's mode is always iptables. There's a default, but a cluster can be configured to use something else. Check kube-proxy's settings in your own cluster.
• Expecting everything to go down if apiserver crashes. Running Pods keep serving, only management breaks: no new Pods get created, and failed ones don't get replaced.

What you'll need for the tasks

• Both labs bring up a kind lab cluster on your laptop. You need Docker, kind v0.31.0 or newer, and kubectl.
• Commands: kubectl get pods -A, kubectl get nodes -o wide, kubectl get daemonsets -n kube-system, kubectl describe pod <name>, the -v=6 flag.
• Lab 2 will have you fix a Pod's YAML manifest and apply it with kubectl apply -f, like in the YAML lesson.

The next lesson covers the Pod itself: what's inside it, why containers get grouped into one Pod, and how kubelet assembles it.