HTTP and HTTPS

Why HTTP

Almost everything a DevOps engineer deals with speaks HTTP: sites, APIs, health checks, webhooks, image registries, the Kubernetes API. When "the site isn't loading," there's always one first question: what did the server answer, and who actually answered. To answer that, you need to be able to read a request and a response, understand status codes, and look at all of it through curl.

The request/response model

HTTP works like a conversation taking turns: the client asks, the server answers. In HTTP/1.1 there are no parallel streams within one connection, and the server never speaks first (in HTTP/2 requests are multiplexed, see below).

Request:

POST /api/login HTTP/1.1
Host: example.com
Content-Type: application/json
Authorization: Bearer eyJhbG...
Content-Length: 39

{"email":"a@b.com","password":"secret"}

Response:

HTTP/1.1 200 OK
Content-Type: application/json
Set-Cookie: session=abc123; HttpOnly; Secure
Content-Length: 27

{"token":"eyJhbGciOiJI..."}

The structure is the same: a start line (method and path, or status), then headers, a blank line, body. Content-Length counts the body's bytes: in the request example it's exactly 39.

But before the client can send this request, TCP has to open a connection. That's three packets between client and server:

If this is HTTPS, a TLS handshake follows right after TCP, and only then does the client send the HTTP request and get a response. The three packets SYN/SYN-ACK/ACK cost about one RTT (round-trip time) before the first byte of data, and that happens for every new TCP connection.

Methods

Method	What it does	Idempotent	Has a body
GET	read a resource	yes	not conventional¹
POST	create / any action	no	yes
PUT	replace the whole resource	yes	yes
PATCH	partially update	no²	yes
DELETE	delete	yes	rarely
HEAD	like GET, but no body (headers only)	yes	no
OPTIONS	which methods are allowed (used by CORS)	yes	no

Idempotency means a repeated call leaves the server in the same state as the first call did. This matters for retry logic: GET and PUT can be safely retried, POST can't (you might accidentally create two orders).

¹ The standard doesn't forbid a body on GET (RFC 9110, section 9.3.1), but there's no point to it, and servers and proxies often ignore or reject it, so this isn't done in practice. ² PATCH is not idempotent by definition (RFC 5789), but a specific patch can be designed to be idempotent (for example, "set field to X").

Statuses

The first digit sets the class:

• 1xx: informational, rarely seen (100 Continue).
• 2xx: success.
• 3xx: redirect.
• 4xx: client error (it sent something wrong).
• 5xx: server error (it failed to handle the request).

You need to know the most common ones by heart:

Code	When
200 OK	standard success
201 Created	a new resource was created (after POST)
204 No Content	success, no response body (after DELETE)
301 Moved Permanently	permanent redirect (the browser caches it)
302 Found	temporary redirect
307 / 308	temporary and permanent redirect where the client must repeat the same method (POST stays POST)
304 Not Modified	your copy is fresh, nothing new to send
400 Bad Request	the request is invalid, fix it
401 Unauthorized	not authenticated, log in
403 Forbidden	authenticated, but no permission
404 Not Found	the resource doesn't exist
409 Conflict	a state conflict (name taken, version is stale)
422 Unprocessable Content	syntactically fine, semantically not (called Unprocessable Entity in older RFCs)
429 Too Many Requests	the client hit a rate limit, needs to wait (how long is often in the Retry-After header)
500 Internal Server Error	the server crashed while handling the request
502 Bad Gateway	the proxy didn't get a proper response from the upstream
503 Service Unavailable	the service is temporarily unavailable
504 Gateway Timeout	the proxy didn't get a response from the upstream in time

The Location header says where a redirect leads. The text after the code (Found, Moved Temporarily) isn't read by clients: nginx, for example, writes 302 Moved Temporarily for a 302. Only the number matters.

502/503/504 are the ones DevOps sees most. Learn to tell them apart:

• 502: the proxy (nginx, Caddy) couldn't get a response from the backend. Most often the backend isn't accepting connections: it crashed, isn't listening on that port, or the proxy is pointed at the wrong address. It also happens when the backend drops the connection or answers with something that isn't HTTP.
• 503: there's nobody to serve the request right now, or it's not allowed. The service itself answers this way (overload, planned maintenance), but so does the proxy, for example when it has no live backends left.
• 504: there's a connection to the backend, but no response within the proxy's timeout. In nginx that's proxy_read_timeout, 60 seconds by default (documentation).

The main clue in cases like this is in the proxy's error log. For a 502 nginx logs connect() failed (111: Connection refused) while connecting to upstream, for a 504 it logs upstream timed out (110: Operation timed out) while reading response header from upstream. Next to it is the upstream address the proxy actually tried.

Headers you'll run into

Header	What it's for
Content-Type	the body's format: application/json, text/html
Content-Length	the body's size in bytes
Authorization	token / basic auth / Bearer
WWW-Authenticate	on a 401: which method the server wants you to authenticate with
Cookie / Set-Cookie	session and state
User-Agent	who the client is (browser, bot, curl)
Location	where to go on a redirect (3xx)
Cache-Control	how the client and proxy should cache: max-age=600 means "the response is fresh for 10 minutes"
ETag / If-None-Match	conditional request: return only if it changed
Host	which domain is being requested (nginx uses it to pick the site when one IP serves many)
X-Forwarded-For	the client's original IP, when there's a proxy

HTTPS

HTTPS is HTTP wrapped in TLS. What it adds:

• port 443 instead of 80;
• encrypted traffic: the provider sees the server's IP and the domain (the client sends it in plaintext in the SNI), but not the path, parameters, or body;
• a certificate: proof that you're talking to the real example.com, not to a MITM.

Certificates are free today (Let's Encrypt), so a production site on plain HTTP without TLS is rare. Caddy issues and renews the certificate automatically (documentation), Traefik can do it too through ACME. Just point it at a domain.

TLS details (the handshake, versions 1.2 and 1.3, exactly how the encryption works) are a separate, big topic. At the application level, you work with HTTPS the same way you work with plain HTTP.

HTTP/1.1, 2 and 3

HTTP/1.1 is text-based. One TCP connection carries one request/response at a time. To load a page with 50 resources, the browser opens up to six parallel connections per host and still waits.

HTTP/2 is binary and multiplexes dozens of requests at once in a single TCP connection. Headers are compressed (HPACK). Server push was in the spec, but Chrome turned it off in 2022 (Chrome for Developers). 103 Early Hints (a separate feature, not a direct replacement) partly solves a similar problem: the server gets to hint which resources the client will need soon, before the main response.

HTTP/3 is built on top of QUIC, and QUIC runs over UDP, not TCP. Losing one packet in TCP stalls every stream on the connection (head-of-line blocking); in QUIC, only its own stream stalls. That's why HTTP/3 is noticeably better on mobile and unstable networks. Browsers support it, but infrastructure (proxies, load balancers, firewalls with UDP/443 closed) doesn't support it everywhere.

What matters for DevOps: nginx and Caddy both support HTTP/2 and HTTP/3. In Caddy, HTTP/3 is on by default; in nginx it's available since version 1.25 and turned on with the listen ... quic and http3 directives (documentation). You can tell a server offers HTTP/3 from the alt-svc: h3=":443" response header. To check the version: curl -v --http2 prints using HTTP/2. curl --http3 only works with curl builds that support HTTP/3: curl on macOS and on Alpine 3.20 answer the installed libcurl version does not support this.

CORS: why "it works in curl, not in the browser"

CORS (Cross-Origin Resource Sharing) is a browser mechanism. Without it, the browser won't let a page's script from one origin (say https://app.example.com) read API responses from another origin (https://api.example.com). A server can explicitly allow this access through headers.

Here's how it shows up for a user: the front end does fetch("https://api.example.com/me"), and the browser console shows a red error like "No 'Access-Control-Allow-Origin' header". But curl https://api.example.com/me from the terminal works fine: CORS is enforced by the browser, not the server.

How a server allows it:

• For a "simple" request, it returns the header Access-Control-Allow-Origin: https://app.example.com. A simple request: GET, HEAD, or POST whose Content-Type is only application/x-www-form-urlencoded, multipart/form-data, or text/plain, with no custom headers.
• For everything else (POST with JSON, custom headers like Authorization, methods like PUT, DELETE), the browser first sends a preflight, an OPTIONS request. The server must answer with a set of Access-Control-Allow-* headers: which origin is allowed, which methods, which headers, whether cookies can be sent (Access-Control-Allow-Credentials: true).

Common production mistakes:

• A request with cookies got Access-Control-Allow-Origin: *. For requests with credentials, * doesn't work: the browser requires a specific origin plus Access-Control-Allow-Credentials: true (MDN). Result: the browser won't hand the response to the script, and the user looks "logged out".
• The preflight forgot to allow a needed header (Authorization, Content-Type), and the browser blocks the actual request.
• In nginx, add_header without the always parameter only fires on success codes and redirects (200, 201, 204, 206, 301, 302, 303, 304, 307, 308) (documentation). On a 401 there are no CORS headers, and the user sees a CORS error instead of the real 401.

curl: see it all with your own eyes

curl -v https://example.com          # all headers, the handshake and the response
curl -i https://example.com          # response together with headers
curl -I https://example.com          # headers only (a HEAD request)
curl -H "Authorization: Bearer X" ... # add a header
curl -X POST -d '{"a":1}' \
     -H "Content-Type: application/json" \
     https://api.example.com/x
curl --json '{"a":1}' https://api.example.com/x  # curl 7.82+, shorter
curl -L https://example.com          # follow redirects
curl -o file.zip https://...         # save the body to a file
curl -s -o /dev/null -w '%{http_code}\n' https://example.com  # response code only
curl --resolve host:443:192.0.2.10 https://host/   # override DNS resolution

Without -L curl doesn't follow redirects: it shows the 3xx response itself with the Location header. That's useful when you need to see exactly where and with what code the server redirects.

curl -v shows the connection, TLS, and headers in both directions. Lines starting with * are curl's own diagnostics, > is the request, < is the response. Shortened output:

$ curl -v https://example.com
* Host example.com:443 was resolved.
...
*   Trying 8.47.69.0:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
...
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=example.com
...
*  SSL certificate verify ok.
...
> GET / HTTP/2
> Host: example.com
> User-Agent: curl/8.11.1
> Accept: */*
>
< HTTP/2 200
< content-type: text/html; charset=utf-8
...
< alt-svc: h3=":443"; ma=86400

You can see the whole chain: the TCP connection, the TLS handshake with its version and cipher, agreeing on HTTP/2 (ALPN: server accepted h2), the certificate check, then the HTTP request and response itself. In HTTP/2, header names are written in lowercase. example.com's IP addresses change over time, yours will be different.

Common mistakes

Confusing 502 and 504. 502 means "couldn't reach the backend" (or it answered with garbage), 504 means "reached it, but didn't get a response in time". They're fixed differently: for the first, you look at where the proxy is actually connecting and whether the backend is alive; for the second, you look at how long the backend takes to respond and how long the proxy is willing to wait.

Looking at the code after a redirect. The browser and curl -L follow Location themselves, so you only see the final 200. To see the redirect itself, request without -L and look at the response's first line and Location.

Confusing 401 and 403. 401 means "who are you?": you need to log in, the server suggests how in WWW-Authenticate. 403 means "I know who you are, but no": logging in won't help, you're missing permissions.

Retrying POST. A timeout on POST doesn't mean the action didn't happen. Retrying without an idempotency key can create a second order.

Debugging CORS through curl. curl doesn't check CORS rules, it doesn't care. Look at the Access-Control-* headers in the response and the Network tab in the browser, including the OPTIONS request.

What they'll ask you in an interview

Short answers you should be able to say out loud:

• "How does 401 differ from 403?" 401: "who are you?", not authenticated, go log in. 403: "I know who you are, but you're not allowed here", missing permissions.
• "How does 502 differ from 504?" 502: the proxy got no response from the backend (it crashed, isn't listening on the port). 504: there's a connection, but the backend didn't answer within the timeout.
• "What's an idempotent method?" A repeated call leaves the server in the same state as the first call did. GET, PUT, DELETE are idempotent, POST and PATCH aren't by default (though a specific PATCH can be designed to be idempotent). That's why POST can't be safely retried.
• "How does HTTP/2 differ from HTTP/1.1?" One TCP connection carries several requests and responses at once (multiplexing), and they don't wait on each other. Headers are compressed (HPACK), the protocol is binary, not text. The browser doesn't need to open six parallel connections.
• "What is CORS, and why preflight?" A browser-side protection: a page's script from one origin can't read API responses from another origin without the server's permission. For "non-simple" requests, the browser first sends an OPTIONS preflight, and the server answers which methods, headers, and origin are allowed. curl doesn't apply this rule, it lives in the browser.
• "What is HTTPS at a high level?" The same HTTP, wrapped in TLS. Port 443; the path, headers, and body are encrypted; the domain is usually visible in the SNI. The certificate proves you're talking to the real server. At the application level, you work with it like plain HTTP.

What you'll need for the tasks

• Docker 20.10 or newer with Compose v2, bash, curl.
• curl -i and curl -I: see the response code and headers; without -L curl doesn't follow redirects (the "curl" section).
• 3xx codes and the Location header, the difference between 401 and 403 (the "Statuses" section).
• Cache-Control: max-age=N (the "Headers you'll run into" section).
• The difference between 502 and 504, and what the nginx error log says about them (the "Statuses" section).
• docker compose -f setup/docker-compose.yml logs <service> and exec <service> <command>: logs and commands inside the lab containers.

In the next lesson we'll go over the toolset used to fix the network when something breaks.