Retries, timeouts, and circuit breakers

Why this matters

A service almost always depends on others: a database, a queue, someone else's API. Any of them can respond slowly or not respond at all. If the service isn't ready for that, one slow neighbor brings down the whole system.

Three techniques make a service resilient. A timeout limits how long it waits. A retry repeats a request that failed because of a temporary problem. A circuit breaker stops calling a dependency that's down. Set up wrong, each of them makes things worse. Below is how to use them, and where they break the system.

Timeout: don't wait forever

A timeout limits how long to wait for a response. Without one, a request to a hung dependency hangs there, holding a connection and a handler thread. These hanging requests pile up until the service hits its own limits: the number of threads, connections, or file descriptors (see the lesson on descriptors).

Many HTTP clients wait forever by default. In Python, requests without a timeout parameter never aborts the request, and urllib behaves the same way unless a global socket timeout is set. curl has no overall timeout by default, only a timeout for establishing the connection (300 seconds). That's why you set a timeout explicitly:

curl -sS --max-time 2 http://127.0.0.1:8080/

urllib.request.urlopen("http://payments/charge", timeout=2)

Pick the timeout based on the dependency's real response time. Too short and it cuts off normal requests. Too long and it doesn't help: the thread is still busy for a long time, and under load the handlers run out. The most common mistake here is simple: an external call with no timeout at all. Sooner or later it hangs the service.

Retry: repeat carefully

A retry repeats a request that failed because of a temporary problem: a timeout, a dropped connection, a 503 Service Unavailable or 429 Too Many Requests response. Errors like 400 or 404 aren't worth retrying: the response will be the same the second time. Retries need care, otherwise they make things worse.

The rules are these.

• Don't blindly retry a non-idempotent request. An idempotent request can be made several times with the same outcome as making it once. By the HTTP standard, GET, HEAD, PUT, and DELETE are idempotent, POST is not. Retrying a POST after a timeout can create a second order: the first request may have gone through and run, the response just didn't make it back in time. The next lesson, on idempotency, covers how to make a POST retry safe.
• Backoff with jitter. Don't retry right away or at a fixed interval. Use a growing pause (exponential backoff: 1 s, 2 s, 4 s) plus a random addition (jitter). Otherwise a thousand clients retry at the same second and finish off a service that's already shaky. If the service sent a Retry-After header, take the pause from it.
• Limit the number of attempts. Unlimited retries turn into the same avalanche.

attempt 1 → fail → wait 1s + rand
attempt 2 → fail → wait 2s + rand
attempt 3 → fail → pass the error up

Retries multiply along a chain. If service A makes 3 attempts to B, and B makes 3 attempts to C, one user request turns into 9 requests to C. Retrying usually belongs at one layer, not at every layer.

Circuit breaker: stop hammering it

If a dependency is down, retries to it are pointless and just burn resources. A circuit breaker tracks the error rate. Once it crosses a threshold, the breaker opens: for a while it stops letting requests through to that dependency and returns an error or a fallback right away, without spending a timeout on every call.

After a pause, the breaker moves to half-open and lets through one trial request. If it passes, the dependency came back, and the breaker closes again. If the trial request fails, the breaker opens again for the next pause. This saves you from a cascade: a down database doesn't drag down every service that calls it.

You usually don't write the breaker by hand. You take a ready one from a library, for example Resilience4j in Java. It has the same CLOSED, OPEN, and HALF_OPEN states, and you set the threshold and the pause in its configuration.

Common mistakes

• A call with no timeout. The client waits forever by default, a hung dependency eats up all of the service's threads, and it stops responding even to its own health check.
• Retrying POST after a timeout. A timeout doesn't mean "it didn't run." Each attempt can create an order of its own.
• Retrying without a pause. All the clients retry in sync and finish off a service that's trying to recover.
• Retries at every layer. Attempts multiply, and the load on the bottom service grows several times over.
• A timeout shorter than the usual response time. Normal requests get cut off, and retries on top double the load.

What to remember

Put a timeout on every external call: without one, a hung dependency hangs the service itself. A retry repeats only temporary errors, with a growing pause and random jitter, otherwise clients finish off the service with a synchronous flood.

Don't blindly retry a non-idempotent call (POST). A circuit breaker opens at a high error rate and returns a failure right away, instead of letting a down dependency drag everything else with it.

The next lesson covers idempotency in APIs: how to make retries safe.

What you'll need for the tasks

• Docker with Compose v2, curl, and bash.
• docker compose -f setup/docker-compose.yml up -d after editing a service's environment:, and docker compose ... logs <service>.
• curl -i and curl -w '%{http_code} %{time_total}\n' to see the response code and how long it took.
• Reading a bit of Python code: where a service gets its timeout from and which requests it retries.