Reverse proxy: nginx and Caddy

Why

The application listens on something like 127.0.0.1:8080 and can serve HTTP. Exposing it to the internet directly is awkward: it doesn't handle TLS, it can't split several domains across different services, and you'd have to wire a certificate into every service separately. So you put a reverse proxy in front of it: nginx or Caddy. The proxy listens on port 443 from the outside, and it forwards requests to the internal application.

It's called "reverse" because the proxy sits on the server side and represents the application to clients. A regular (forward) proxy does the opposite: it represents the client to servers, like a corporate proxy that employees go through to reach the internet.

Concepts

• Reverse proxy — a server that accepts client requests and forwards them to internal applications (upstreams, backends).
• Upstream — the application behind the proxy that a request goes to. In nginx its address is set by proxy_pass.
• TLS termination — the proxy accepts HTTPS, decrypts it, and then talks to the application over plain HTTP.
• Virtual host — a config block for one domain. In nginx this is a server with its own server_name.

TLS termination

Terminating TLS is the proxy's most common job. The client opens an HTTPS connection with the proxy, the proxy decrypts the request and passes it to the application over HTTP inside a trusted network.

Client --HTTPS--> [nginx :443] --HTTP--> [app :8080]

The application doesn't need to know about certificates and encryption. A certificate from Let's Encrypt is wired in one place, the proxy config, instead of in every service.

In nginx, three lines inside server take care of this:

listen 443 ssl;
ssl_certificate     /etc/nginx/certs/site.crt;
ssl_certificate_key /etc/nginx/certs/site.key;

Routing by domain and path

One proxy serves several sites on the same IP. For HTTP it looks at the Host header; for HTTPS it also looks at SNI, the server name the client sends at the start of the TLS handshake. By that name it picks a server and routes the request further:

server {
    listen 80;
    server_name api.example.com;
    location / {
        proxy_pass http://127.0.0.1:8080;
    }
}

server {
    listen 80;
    server_name shop.example.com;
    location / {
        proxy_pass http://127.0.0.1:9090;
    }
}

A request to api.example.com goes to port 8080, to shop.example.com to 9090. You can also route by path: location /api/ sends to one application, location / to another.

You can check the routing without DNS: curl lets you set any Host.

curl -H 'Host: shop.example.com' http://127.0.0.1/

The default server

What happens to a request whose Host doesn't match any server_name? Say, someone hits the server by bare IP. nginx sends it to the default server for that port. If you don't set one explicitly, the first server in the config becomes the default. In the example above that's api, and a request by bare IP silently ends up in the API.

You set the default server explicitly with the default_server parameter on the listen directive. It's usually a stub that answers with an error and proxies nowhere. Details: How nginx processes a request.

Caddy

Caddy does the same thing more concisely:

api.example.com {
    reverse_proxy 127.0.0.1:8080
}

If the site's address is a public domain, Caddy gets a certificate itself (Let's Encrypt or ZeroSSL) and redirects HTTP to HTTPS on its own. For that, the domain's DNS record has to point at the server, and ports 80 and 443 have to be reachable from outside (Automatic HTTPS).

Headers: who the real client is

After proxying, the application sees a connection from the proxy, not from the real client. If you configure nothing, every request in the application's logs shows the proxy's IP, and the scheme shows http even though the client came in over https. On top of that, nginx by default replaces Host with the address from proxy_pass ($proxy_host), so the application sees something like app:8000 instead of the domain.

To pass the application the truth, the proxy adds a few headers:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

• Host — which domain the client requested. $host doesn't include the port. If the site runs on a non-standard port and the application builds links with the port in them, use $http_host instead: the client's Host header as is.
• X-Forwarded-For — the chain of addresses: the client and every proxy up to this one. $proxy_add_x_forwarded_for takes what the client sent and appends $remote_addr, the address of whoever connected to nginx.
• X-Forwarded-Proto — which scheme the client used (https). Without it, an application that requires HTTPS sees http and redirects the client to HTTPS again and again: an infinite redirect.
• X-Real-IP — a single client address, with no chain.

Caddy's reverse_proxy sets X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host on its own, and passes the client's Host through unchanged (reverse_proxy, Defaults).

Who to trust

The X-Forwarded-For header can also be sent by the client itself, with any address inside. nginx doesn't strip it, it appends its own address at the end. So you can only trust the entries your own proxy added: the right-hand side of the chain. The leftmost entry can be made up.

X-Forwarded-For: 1.2.3.4, 203.0.113.7
                 ^ sent by the client  ^ added by nginx: the real connecting address

The application should accept these headers only from its own proxy, otherwise they can be forged from outside. Caddy ignores incoming X-Forwarded-* from clients by default; you can trust them only from the addresses listed in trusted_proxies.

Common mistakes

• No X-Forwarded-Proto with TLS on the proxy. An application with forced HTTPS ends up in an infinite redirect, and the browser shows "too many redirects".
• No default server. Requests with an unfamiliar or empty Host land in the first server in the config, meaning some application ends up handling them.
• Host isn't passed through. The application sees app:8000 from proxy_pass and builds links and redirects to the internal address.
• The application trusts the first address in X-Forwarded-For. A client can put anything there and dodge a ban or a rate limit.
• The config was edited but not reloaded. nginx reads the config at startup and on nginx -s reload. Check the syntax before reloading: nginx -t.

What you'll need for the tasks

• server, listen and its default_server parameter, server_name, location, proxy_pass, return <code>;
• proxy_set_header and the variables $host, $http_host, $remote_addr, $proxy_add_x_forwarded_for, $scheme;
• checking and reloading the config: nginx -t, nginx -s reload inside the container through docker compose exec;
• curl -H 'Host: …', curl -k for a self-signed certificate, curl -i to see the response code and Location;
• docker compose logs <service>.

The next lesson covers load balancing, when more than one identical service sits behind the proxy.