Designing robust and predictable APIs with idempotency

Networks fail at a constant background rate. Any call between two machines can fail in three distinct ways. First, the connection fails before it ever reaches the server. Second, the call reaches the server but fails partway through, while the server is doing the work. Third, the work succeeds but the connection breaks before the client hears the result. The first case is easy: nothing happened, so the client can safely retry. The other two are the hard ones. In both, the operation may or may not have taken effect, and from where the client sits those two outcomes look identical - so it has no safe way to know whether retrying will repeat the work or not.

For most operations this ambiguity is a mild annoyance. For a payments API it is the worst case. In an ambiguous failure, the client has to guess. If it retries and the charge had in fact already succeeded, it double-charges the customer. If it decides not to retry and the charge had in fact not gone through, it drops real revenue and leaves the customer without what they paid for. Without some scheme agreed between client and server ahead of time, every transient network hiccup forces a choice between those two bad outcomes. And there is no way to choose correctly, because the very information needed to choose is what got lost.

The post is careful to frame how low the bar is for this problem to apply. A distributed system here means as little as two computers passing messages over a network - the Stripe API plus any one server calling it already qualifies. There is no scale threshold below which the ambiguous-failure problem goes away; it is a property of the network itself, present from the very first remote call, not something that appears only once traffic grows large.

The first layer is making endpoints idempotent wherever the operation's meaning allows it - idempotent meaning the endpoint can be called any number of times while the underlying change happens only once. HTTP already builds this in. Under the standard that defines HTTP methods (RFC 7231), PUT and DELETE are defined as idempotent. A PUT that fully specifies the target resource (the post's example is a DNS provider creating a CNAME record) can simply be retried until it verifiably succeeds. The server treats a duplicate as a no-op - it sees the resource already exists and just returns success. For this class of operation, a retry loop is the whole answer; no extra machinery is needed.

For operations that must happen exactly once - the classic example is charging a customer - idempotency keys carry the guarantee instead. The client generates a unique ID for the logical operation and sends it with the request; the server ties that key to the state of the request on its side. Stripe implements this on all mutating (POST) endpoints through the Idempotency-Key header: send the charge again with the same key, and the customer is charged only once.

The key resolves each of the three failure cases differently. On a connection failure, the retry is the first time the server has seen the key, so it just processes it normally. On a mid-operation failure, the server picks up the interrupted work and carries it through. The post is explicit that the mechanics here depend heavily on implementation. If the interrupted attempt was cleanly rolled back by an ACID database (one that guarantees a partial operation leaves no trace), the retry can safely re-run the whole thing. Otherwise, the server has to recover the partial state and continue from where it stopped. On a response failure - the work finished but the client never heard - the server simply replays the saved result of the completed operation.

The final layer is what the post calls being a good distributed citizen. A failure might be a brief blip, or it might be a server in the middle of an outage - and in the second case, persistent retries pile load onto a server exactly when it can least afford it. So clients should back off exponentially, waiting proportionally to 2^n as failures repeat, and add random jitter to each wait. The reason jitter matters: if one incident knocks out many clients at the same instant, plain backoff leaves them all on the same schedule, so their retries land in synchronized waves that hammer the recovering server - the thundering herd. Randomizing each client's waits spreads those retries out. Stripe's own Ruby client library ships the whole combination by default: automatic retries with an idempotency key, increasing backoff, and jitter.

WHY RETRIES NEED JITTER