Load balancing (L4 and L7)

Why more than one server

A single server hits a ceiling: it can't handle the load, or it goes down entirely, and the site is dead. The fix: run several identical copies of the app and spread requests across them. A load balancer handles the distribution.

This gives you two things at once. Scaling: more replicas, more total load you can handle. Fault tolerance: one replica goes down, the load balancer stops sending it traffic, and the rest keep working.

Concepts

• Load balancer — a server that accepts requests and distributes them across several identical copies of the app.
• Upstream — the group of replicas behind the load balancer. In nginx this is an upstream block with several server lines.
• Load balancing algorithm — the rule that picks a replica for the next request.
• Health check — a check of whether a replica is alive. Based on the result, a replica is pulled out of rotation or put back in.
• Stateless app — a replica that doesn't keep the client's state (session, cart) to itself. Any replica can serve any request.

What this looks like in nginx

The load balancer is often combined with the reverse proxy from the previous lesson. In nginx, replicas are listed in the upstream block, and proxy_pass gets the group name instead of an address:

upstream shop {
    server 10.0.5.10:8080;
    server 10.0.5.11:8080;
    server 10.0.5.12:8080;
}

server {
    listen 80;
    location / {
        proxy_pass http://shop;
    }
}

Without extra settings, nginx hands out requests in turn (round-robin). The server line has a weight= parameter: a replica with weight=3 gets three times as many requests as a replica with the default weight of 1.

Distribution algorithms

• Round-robin — each request goes to the next server in line. The simple default.
• Least connections — goes to whoever currently has the fewest active connections. Better when requests vary in length. In nginx, the least_conn; directive inside upstream.
• IP hash — a client always lands on the same server based on its IP. In nginx, the ip_hash; directive. The key is the first three octets of the IPv4 address, so every client from the same /24 network (for example, behind one office NAT) lands on one replica.
• Hash by key — the same idea, but you pick the key yourself: hash $request_uri consistent; sends the same URL to the same replica every time. The consistent parameter turns on consistent hashing: adding or removing a replica only moves a small fraction of the keys.

Without pinning, a client can land on a different replica on every request. That's why replicas are made stateless: sessions and data live in a shared place (a database, Redis), not in a specific server's memory. Then any replica can serve any request.

Pinning a client to a replica (sticky sessions) looks like a quick fix, but it doesn't remove the problem. The replica goes down, and all its sessions disappear with it.

Health checks

The load balancer needs to know which replicas are alive. There are two kinds of checks.

Active. The load balancer itself periodically pings each replica, usually a lightweight /healthz endpoint that quickly responds 200 OK if everything's fine. This is how cloud load balancers, HAProxy (option httpchk), and Kubernetes readiness probes work. In nginx, active checks (the health_check directive) exist only in the commercial NGINX Plus.

Passive. The load balancer watches the responses sent to real clients. Free nginx only knows how to do this. An attempt counts as failed if it matches the conditions of the proxy_next_upstream directive. By default that's error timeout: the connection couldn't be made, it dropped, or the replica didn't respond in time.

upstream shop {
    server 10.0.5.10:8080 max_fails=1 fail_timeout=10s;
    server 10.0.5.11:8080 max_fails=1 fail_timeout=10s;
}

The values in the example match the defaults. If max_fails failed attempts pile up within fail_timeout, nginx stops sending requests to that replica for the fail_timeout duration, then tries it again. A request that hits a failed attempt gets retried by nginx on the next replica, and the client gets a normal response.

Important detail: by default, an HTTP error response doesn't count as a failed attempt. A replica responds 503, and nginx faithfully passes that 503 to the client. To make such responses trigger a retry on another replica and pull the replica out of rotation too, you list the codes explicitly, for example proxy_next_upstream error timeout http_502;. Which codes to include depends on what the unhealthy replica actually returns.

L4 and L7

You can load balance at different layers (same as proxying).

L4 load balancer works at the TCP/UDP level. It only looks at IPs and ports, and doesn't parse the request's content. Fast, but it can't route by URL or domain. Example: a cloud Network Load Balancer (AWS NLB).

L7 load balancer works at the HTTP level. It sees the URL, headers, and cookies, and can route smartly: /api/ to one set of servers, /static/ to another; keep a client on one replica by cookie. Examples: a cloud Application Load Balancer (AWS ALB), nginx in the http block.

nginx and HAProxy can do both. In nginx, L4 balancing lives in a separate stream block at the top level of nginx.conf, next to http, not inside it:

stream {
    upstream tcp_app {
        server 10.0.5.10:5432;
        server 10.0.5.11:5432;
    }
    server {
        listen 5432;
        proxy_pass tcp_app;
    }
}

Here nginx spreads out TCP connections and doesn't know what's inside: HTTP, PostgreSQL, or something custom. In HAProxy, the same thing is toggled with the mode tcp or mode http line.

The rule is simple. If you only need to spread out TCP connections, L4 is enough. As soon as you need to decide based on HTTP content, use L7.

Common mistakes

• Expecting nginx to pull out a replica that responds 500/503 on its own. Without http_5xx codes in proxy_next_upstream, it counts that response as successful and passes it to the client.
• A stopped replica and a long timeout. If a replica's address stops responding entirely, nginx waits proxy_connect_timeout for the connection, 60 seconds by default. The first client after the replica goes down waits a minute before nginx switches over. For an internal network, the connection timeout is set to a few seconds.
• POST isn't retried. nginx won't send a request with a non-idempotent method (POST, PATCH, LOCK) to the next replica if it already passed it to the first one: the order might get created twice. This is the default behavior, changed by the non_idempotent parameter, and it's worth turning on only for operations that are safe to repeat.
• Each worker keeps its own counters. If the upstream has no zone directive, every nginx worker process keeps its own round-robin queue and its own failure counters. So request distribution doesn't look strictly round-robin, and each worker finds the unhealthy replica on its own.
• Replicas were recreated, but nginx still sends requests to the old addresses. nginx resolves names from upstream to IPs at startup and on nginx -s reload. In Docker, a recreated container can get a different address, and requests for app1 end up at the wrong replica or nowhere. Reload nginx after recreating replicas.
• Sticky sessions instead of stateless. ip_hash "fixes" lost sessions, but the client ends up pinned to one replica, and behind NAT, the whole /24 network lands on one replica.

What you'll need for the tasks

• the upstream block, server lines and their weight, max_fails, fail_timeout, down, backup parameters;
• proxy_pass http://<upstream name>, proxy_next_upstream and its values error, timeout, http_502, http_503, http_504;
• ip_hash, hash, least_conn, and the difference between pinning a client to a replica and shared session storage;
• a service's environment variables in docker-compose.yml and recreating a service with docker compose up -d;
• from earlier lessons in the module: listen 443 ssl, ssl_certificate with fullchain.pem, return 301, proxy_set_header with $http_host, $proxy_add_x_forwarded_for, $scheme, checking with curl --cacert and --resolve;
• nginx -t, nginx -T, nginx -s reload through docker compose exec, docker compose logs.

In the next tracks this connects to Kubernetes: a Service balances at L4, while Ingress and the Gateway API provide L7 routing inside the cluster.