HorizontalPodAutoscaler

How HPA calculates the replica count it needs, why it stays silent without resources.requests, what the behavior block does, and how to read what HPA is doing right now.

Why you need HPA

Load on a service changes: a peak during the day, quiet at night, a spike during a sale. Keeping enough replicas for the peak is expensive, keeping only enough for the average load is risky. HPA handles this automatically.

HorizontalPodAutoscaler (HPA) is a controller that automatically changes a Deployment's replica count based on load. A Deployment is an object that keeps the wanted number of identical Pods running. A Pod is the smallest unit of execution; a container runs inside it. A replica is one of the identical Pods in the set.

HPA's goal is to keep average resource usage close to a set percentage. If each replica is using 90% CPU with a target of 50%, HPA adds replicas. If usage drops to 10%, it removes some.

HPA only changes the replica count of a single workload. Changing a Pod's requests is the job of VerticalPodAutoscaler. Adding and removing nodes is the job of Cluster Autoscaler. HPA is the most commonly used of the three.

How the loop works

HPA lives inside kube-controller-manager. Every 15 seconds it polls metrics-server, calculates current usage, and compares it with the target. metrics-server is the component that collects current resource metrics from the nodes. After the calculation, HPA changes spec.replicas on the target Deployment, and the usual rollout mechanism takes over from there.

The interval is set by the --horizontal-pod-autoscaler-sync-period flag on kube-controller-manager, 15 seconds by default.

metrics-server is installed into a cluster separately. Cloud clusters usually already have it, kind doesn't. Without it, kubectl top and CPU-based HPA don't work.

Minimal YAML

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: cpu-burner
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: cpu-burner
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

The same setup with one command:

kubectl autoscale deployment cpu-burner --min=1 --max=5 --cpu=50%

The --cpu=50% flag appeared in newer kubectl versions. The older --cpu-percent=50 still works in kubectl v1.36, but it's marked deprecated and prints a warning. The command takes the HPA's name from the Deployment's name.

Field breakdown

scaleTargetRef: the object the HPA manages. Usually a Deployment, sometimes a StatefulSet. You point it at the controller, not a Pod. HPA looks it up by kind and name in its own namespace. If no such object exists, the server still creates the HPA; the error only shows up in its status.

minReplicas and maxReplicas: the lower and upper bounds on the replica count. maxReplicas is required, minReplicas defaults to 1.

metrics: a list of metrics to base the decision on. There can be several metrics; HPA calculates a replica count from each and takes the maximum.

target.type: Utilization sets the target as a percentage of requests. requests: the amount of resources a Pod asks the scheduler for. The type AverageValue sets the target as an absolute value, for example 200m CPU per Pod.

The decision formula

desiredReplicas = ceil(currentReplicas * currentMetricValue / targetMetricValue)

One Pod is using 100% of requests, the target is 50%. That gives ceil(1 * 100 / 50) = 2. HPA sets 2 replicas. If both Pods keep using 100%, the next tick gives ceil(2 * 100 / 50) = 4. Then HPA hits maxReplicas and stops.

HPA ignores small deviations. If the ratio of the current value to the target differs from 1 by less than 10%, the replica count doesn't change. This is the tolerance, and it saves you from replicas jittering on every tick.

A noticeable amount of time passes between a load increase and new Pods appearing. Metrics arrive with a delay, HPA waits for its own tick, and then the new Pods still need to start and become Ready. That's why HPA doesn't save you from a sharp spike that happens within seconds; a reserve of replicas for that kind of spike comes from minReplicas.

behavior: different speeds up and down

autoscaling/v2 has a behavior block:

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
      - type: Percent
        value: 100
        periodSeconds: 30
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 10
        periodSeconds: 60

stabilizationWindowSeconds is the stabilization window. When scaling down, HPA looks at all its own recommendations over this window and takes the largest one. policies limit the speed: here, no more than doubling within 30 seconds going up, no more than 10% of replicas per minute going down.

By default, HPA scales up immediately: the scaleUp window is 0. Going down, it waits a window of 300 seconds, that is 5 minutes after load drops. This protects against unnecessary churn from short dips. It's better to keep an extra Pod around than to kill it and create it again a minute later.

Fields you didn't specify fall back to their defaults. You can look up their description right in the cluster:

kubectl explain hpa.spec.behavior.scaleDown

requests are required

Without resources.requests, HPA doesn't work on a percentage basis. The TARGETS column will show cpu: <unknown>/50%, and autoscaling stays silent. The reason is that averageUtilization is a percentage of requests, and without them there's nothing to divide by. A single container in the Pod without CPU requests is enough to make the metric for the whole Pod uncalculable.

<unknown> can also show up for another reason: no metrics-server. kubectl describe hpa helps tell the two apart; the reason is written in Conditions and Events.

How to read HPA state

kubectl get hpa

NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: <unknown>/60%   2         5         2          40s

REFERENCE shows what the HPA manages. TARGETS shows the current value and the target. REPLICAS is how many replicas HPA currently sees on its target.

kubectl describe hpa <name> shows the conditions (Conditions), three in total. Right after creation, or on an error, not all of them show up yet.

• AbleToScale: whether the HPA can even read and change its target's replica count. False with reason FailedGetScale means the target wasn't found.
• ScalingActive: whether it can calculate metrics. False means there are no metrics, or nothing to calculate a percentage from; the reason is in the Reason column.
• ScalingLimited: whether the desired count has hit minReplicas or maxReplicas.

Below, in Events, HPA writes what it did and why, for example New size: 4; reason: ....

Related commands

kubectl autoscale deployment <name> --min=1 --max=5 --cpu=50%
kubectl get hpa
kubectl describe hpa <name>           # Conditions and Events with the reasons behind decisions
kubectl get hpa -A                    # all HPAs in the cluster
kubectl explain hpa.spec.behavior

Common mistakes

• No requests. TARGETS shows <unknown>, HPA stays idle.
• No metrics-server. Same symptom, <unknown>, the reason shows up in describe.
• scaleTargetRef points at an object that doesn't exist: a typo, or the Deployment was renamed. The HPA is created without errors, but does nothing, in Conditions AbleToScale False.
• A maxReplicas set too high combined with a memory leak in the app. HPA scales all the way to maxReplicas, and the cluster grinds to a halt. A sane max is required.
• A manual kubectl scale while an HPA is live. A manually set replica count survives only until the HPA's next sync, at most 15 seconds, then HPA sets its own.
• replicas in the Deployment's manifest while an HPA is live. Every kubectl apply of that manifest resets the replica count to the value in the file, and the HPA has to catch up again. The documentation recommends dropping replicas from the manifest once you turn on an HPA.

What you'll need for the tasks

• An HPA manifest in autoscaling/v2, or kubectl autoscale with the --min, --max, --cpu flags.
• The behavior.scaleDown.stabilizationWindowSeconds block.
• kubectl get hpa and the REFERENCE, TARGETS, REPLICAS columns.
• kubectl describe hpa and the AbleToScale, ScalingActive conditions.
• kubectl patch or kubectl edit to edit an HPA.

The tasks are in the labs/ folder: each lab brings up its own kind cluster on your computer, instructions are in its README.md. metrics-server isn't installed in the labs, so HPA doesn't see real load there, but the bounds and settings work the same way as in production.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-hpa     # set up the starting state
./run.sh verify k8s-hpa    # check the current step

In short

• HPA edits a Deployment's spec.replicas based on metrics from metrics-server.
• Formula: ceil(current replicas * current value / target), with a 10% tolerance.
• Without resources.requests there's nothing to calculate the percentage target from, and HPA shows <unknown>.
• It scales up immediately, but waits a 5-minute stabilization window going down.
• behavior in autoscaling/v2 lets you set your own speeds and windows.
• Look for the reason an HPA is idle in kubectl describe hpa, in Conditions.
• A manual kubectl scale while an HPA is live holds only until the next tick.