Designing robust and predictable APIs with idempotency
How Stripe makes APIs safe in an unreliable network: idempotent endpoint design where HTTP semantics allow it, client-supplied idempotency keys where operations must happen exactly once, and disciplined retry behavior — exponential backoff with jitter — so that recovering clients never become the next outage.
Crash the same charge three ways — with and without an idempotency key.
Problem
Networks fail at a constant ambient rate, and any call between two nodes can fail in three distinct ways: the initial connection fails before reaching the server, the call fails midway while the server is doing the work, or the operation succeeds but the connection breaks before the client hears about it. The first case is unambiguous — nothing happened, retry freely. The other two leave the client in genuine uncertainty: it cannot tell whether the operation took effect, so it doesn't know whether retrying is safe.
For most operations this ambiguity is an annoyance. For a payments API it's the worst case: retrying a charge that actually succeeded double-charges the customer, while abandoning a charge that actually failed drops real revenue. Without a coordinated scheme between client and server, every transient network hiccup forces a choice between those two bad outcomes.
The post is careful to frame how low the bar is for this problem to apply: a distributed system here means as few as two computers passing messages over a network. The Stripe API plus any one server calling it already qualifies. There is no scale threshold below which the ambiguous-failure problem disappears.
Solution
The first layer is making endpoints idempotent wherever the operation's semantics allow it — callable any number of times while guaranteeing side effects occur only once. HTTP already encodes this: per RFC 7231, PUT and DELETE are idempotent verbs, and a PUT that fully specifies the target resource (the post's example is a DNS provider creating a CNAME record) can simply be retried until it verifiably succeeds; the server treats duplicates as no-ops and returns success. For this class of operation, the retry strategy is the whole answer.
For operations that must happen exactly once — the canonical example is charging a customer — idempotency keys carry the guarantee instead. The client generates a unique ID for the logical operation and sends it with the request; the server correlates the key with the request's state on its side. Stripe implements this on all mutating (POST) endpoints via the Idempotency-Key header: retry the charge with the same key, and the customer is charged once.
The key resolves each of the three failure cases differently. On a connection failure, the retry arrives as the first time the server sees the key, and it processes normally. On a mid-operation failure, the server picks up the work and carries it through — the post is explicit that the mechanics here are heavily implementation-dependent: if the interrupted attempt was rolled back by an ACID database, the retry can safely re-execute wholesale; otherwise the server must recover state and continue the call. On a response failure — the work succeeded but the client never heard — the server simply replays the cached result of the completed operation.
The final layer is what the post calls being a good distributed citizen: a failure might be a blip, or it might be a server mid-incident, and persistent retries against the latter add load exactly when the server can least afford it. Clients should back off exponentially (waiting proportionally to 2^n across failures) and add random jitter to each wait, because synchronized failure produces synchronized retries — the thundering herd — unless individual schedules are deliberately decorrelated. Stripe's own Ruby client library implements all of it by default: automatic retries with an idempotency key, increasing backoff, jitter.
Tradeoffs
- Idempotency keys move real work onto the server's bookkeeping: it must durably store key-to-state records and consult them on every request, and that storage must be retained long enough to outlive any realistic retry window. The post's response-failure case — replaying the cached result — only works if the result is still there when the retry arrives, so retention policy becomes part of the API's correctness contract, not an operational afterthought.
- The burden of key discipline lands on the client. The client generates the key, decides its scope, and must never reuse a key across logically distinct operations — a client that recycles an old key for a new charge gets the old charge's cached outcome back, silently. The server cannot detect this; correctness now depends on key-generation hygiene in every integrating codebase.
- The hardest failure case — a crash midway through the operation — is named but not solved by the key alone. The post says it directly: recovery behavior is heavily dependent on implementation. An ACID rollback makes wholesale retry safe, but any operation whose side effects span systems that don't share a transaction needs explicit state recovery before the call can continue. The key tells the server that this is a retry; what the server does with that knowledge is where the real engineering lives.
- Good citizenship costs the individual client. Exponential backoff means a client's state can stay divergent from the server's for progressively longer windows — by the fifth failure it's waiting tens of seconds — and jitter deliberately adds more delay variance on top. The client accepts slower personal convergence so that the population of clients doesn't collectively prevent the server from recovering at all.
- The two idempotency regimes don't unify. Naturally idempotent endpoints (PUT/DELETE, fully-specified resources) need no keys; exactly-once operations (POST charges) need the full key machinery. An API offering both carries two different retry contracts, and integrators must know which regime each endpoint lives in — a documentation and API-design burden the post addresses by recommending idempotency be applied liberally and explicitly.
Patterns in this article
- Idempotency Keys
The source of the pattern's canonical form: a client-generated unique ID sent with each mutating request (Stripe's Idempotency-Key header on all POST endpoints), letting the server resolve all three network-failure cases — process fresh, pick up and carry through, or replay the cached result. Same key in, same outcome out.
- Retry with Backoff and Jitter
The post's 'good distributed citizen' section is a compact statement of the whole pattern: exponential backoff (2^n) so persistent failures slow the retry rate, plus random jitter so synchronized failures don't become synchronized retries (the thundering herd). Stripe's Ruby library ships the full combination — retries, keys, backoff, jitter — as default client behavior.
Also solving this
Other systems in behindscale's Ambiguous failure under retry class: