Idempotency in APIs

Why this matters

A client sends POST /orders, the network blips, the response never arrives. The client doesn't know whether the order was created or not, and retries the request. If the server creates a new order on every POST, the retry makes a second one. This is exactly why the previous lesson on retries forbade retrying POST blindly. A timeout on the client cuts off waiting for a response, not the server's work: the order may already exist by then.

The fix for this is called idempotency. An idempotent operation gives the same result no matter how many times it's repeated. One order stays one order, whether the request was retried or not.

Which methods are idempotent on their own

The HTTP standard (RFC 9110) calls GET, HEAD, PUT, DELETE, OPTIONS, and TRACE idempotent. Read twice, replace a resource twice, delete twice: the state on the server ends up the same as after one call.

Idempotency is about the state on the server, not the response. A second DELETE of the same order can return 404, because there's nothing left to delete. The order is still deleted exactly once, and that's fine.

POST is not idempotent: every call creates something new. That's exactly what makes retrying it produce duplicates.

So wherever possible, pick an idempotent method. If the client comes up with the order id, create it through PUT /orders/{id}: a retry overwrites the same order instead of creating a second one.

PUT /orders/ord-42
{ "item": "book", "qty": 1 }

Per RFC 9110, the first such request, the one that created the resource, answers 201 Created, and a retry with the same id replaces it and answers 200 OK or 204 No Content. There is still one order. The id must be unique per operation: generate it once before the first attempt and don't change it on retries.

Idempotency key

When you need POST specifically (the server assigns the id), use an idempotency key. The client generates a unique key per operation and sends it in a header:

POST /orders
Idempotency-Key: 9f1c-44a2-...

{ "item": "book", "qty": 1 }

The server remembers the keys it has processed. When a request arrives with a key it has already seen, the server doesn't create a new order. It returns the result of the first one. A retry is safe: same key, same order. This is how payment APIs work, where a duplicate means money charged twice.

1st request: new key   → create the order, save (key → response)
2nd request: seen key  → return the saved response, create nothing

The Idempotency-Key header isn't a standard yet, just an IETF draft (draft-ietf-httpapi-idempotency-key-header). In practice Stripe and other payment APIs use it, so the header name comes up often.

Keys are stored with an expiration: there's no reason to keep them longer, retries arrive within a short retry window. Stripe lets you delete a key once it turns 24 hours old.

Where to store keys

Keys need to be visible to every copy of the service. If a key lives in a process's memory, a retry that the load balancer sends to a different copy won't find that key and will create a duplicate. This is the same "stateless processes" rule from the lesson on config and 12-factor: shared state lives in external storage.

A common choice of storage for keys is Redis. The SET command with the NX and EX flags writes a key in one step only if it doesn't exist yet, and sets its expiration:

SET idem:9f1c-44a2 "payment 7" NX EX 86400

An OK response means the key is new. A (nil) response means the key already existed. The point is that the check and the write happen in one command: two simultaneous requests with the same key can't both decide the key is new.

Why this goes together with retries

Idempotency and retries work as a pair. A retry, from the previous lesson, repeats a failed request, and idempotency guarantees the retry won't cause harm. Without idempotency, retrying a POST is dangerous. With an idempotency key, you can retry it as calmly as a GET.

The key is generated once per operation, before the first attempt. A new key on every attempt turns the retry into a new operation, and the protection disappears.

The same applies to queue handlers. A message can arrive more than once: with at-least-once delivery, the broker resends a message if it didn't get an acknowledgment (ack), and the ack gets lost when the handler crashes right after doing the work. The handler must be idempotent, or one event gets applied twice. Usually a message has an id, and the handler remembers processed ids the same way the server remembers keys.

Common mistakes

• Key in one copy's memory. Works fine while there's only one copy. A second copy behind the load balancer doesn't see the other's keys, and a retry that lands on it creates a duplicate.
• New key on every attempt. The key gets generated inside the retry loop, and the server sees three different operations instead of one.
• Key without an expiration. The storage grows forever. Set an expiration with margin for the retry window, a day for example.
• Check and write done separately. GET the key, then SET: two simultaneous retries both fail to find the key and both create an order. You need an atomic operation like SET ... NX.
• A non-idempotent queue handler. INCRBY on the balance for every message with no record of processing: redelivery credits the money twice.

What to remember

Retrying a request should not create a duplicate. GET, PUT, DELETE are idempotent on their own. Where possible, create resources through PUT with an id the client came up with.

For POST, use an idempotency key: the server remembers the key in shared storage and returns the result of the first call on a retry. The key is generated once per operation and lives for a limited time.

Idempotency makes retries safe and is required for queue handlers with at-least-once delivery.

This closes out the application patterns module: graceful shutdown, health endpoints, config, resilience, and idempotency. The mini-project in lab 03 brings them all together on one service.

What you'll need for the tasks

• Docker with Compose v2, curl, and bash.
• curl -X PUT and curl -H 'Idempotency-Key: ...', plus the retry flags --max-time and --retry from the previous lesson.
• docker compose -f setup/docker-compose.yml logs <service>, to count how many requests reached the service.
• redis-cli inside the container: GET, SET ... NX EX, TTL, --scan --pattern.
• Reading and editing a small amount of Python code.
• For the mini-project: PID 1 and exec (lesson 1), healthcheck (lesson 2), state outside the process (lesson 3), timeout and retry (lesson 4).