Asked Twice, Done Once: How AWS Builds Idempotent APIs
Launching one EC2 instance takes several calls under the hood (placement, storage, networking, provisioning the VM), and when one fails transiently, the simplest cure is the best one: retry until it succeeds, a pattern so effective AWS bakes default retries into its SDKs. Malcolm Featonby's Builders' Library piece is about the assumption that cure rests on: that retrying a call causes no extra side effects. The dilemma: a customer running a singleton workload (at most one instance, ever) sees its RunInstances call time out, and retrying could launch a second instance. AWS's answer is an idempotent API contract, where idempotent means a call can be repeated with no extra effect: the caller attaches a unique request identifier (EC2's ClientToken), the service records it with every change as one atomic (all-or-nothing) unit, and for that identifier's lifetime it answers any repeat by replaying the original result instead of redoing the work. A repeat with different parameters is rejected, since a changed request means a changed intent.
Run a singleton workload's RunInstances with the response cut mid-flight, and retry your way into two instances — then arm the ClientToken and watch the replay. Break the token's atomicity and duplicate anyway; try a synthetic hash and dedupe a customer's real intent; terminate the instance and meet the late retry that replays success about the dead.
Problem
The post opens with the pattern that makes retries load-bearing at AWS: a complex operation breaks into a controlling process that calls smaller services (an EC2 launch touches placement, EBS volume creation, network interfaces, and VM provisioning), and the controlling process has to drive all of them to a good state. A surprisingly large share of transient faults go away if you just retry, which is why AWS SDKs retry by default on network failures, server-side faults, and rate limiting: it removes a whole category of repetitive error-handling code from every calling service. But the default rests on one simplifying assumption the post names precisely: that the operation can be retried without extra side effects. You would not want a retried EBS-volume call to leave you with two volumes.
The scenario that sharpens that assumption into a dilemma: a customer's provisioning process runs a singleton workload (at most one EC2 instance at any time), and its RunInstances call gets no response. Is the workload running or not? Simply retrying could produce a second one, with serious consequences; not retrying might leave the launch stranded. The escape without a contract is reconciliation (query the system, compare, decide), which the post rejects on two grounds: it is a lot of work to handle a rare edge case, and it still doesn't resolve the uncertainty, because an instance you find might have been created by a different provisioning process. Whether this instance is yours matters, and no amount of looking after the fact can tell you.
Even the obvious shortcut for spotting duplicates fails on intent. A synthetic token (a hash of the request's parameters) treats identical-looking requests as duplicates, which is plausible for two near-simultaneous CreateTable calls but wrong for EC2, where the customer may genuinely want two identical instances. Same parameters is not the same as same intent, and only the caller knows the difference.
Solution
AWS's preferred contract makes the caller state intent directly: a unique caller-provided request identifier (the ClientToken in EC2's API), where two requests from the same caller carrying the same identifier are duplicates by definition. The choice pays off in other ways the post lists: the identifier shows up in CloudTrail logs, so intent is auditable, and the created resource is labeled with it (it shows up when you look the instance up later with DescribeInstances), so a customer can trace any resource back to the request that made it.
On the server side (the AWS service receiving the call, not the customer's code that made it), handling a request means checking whether the identifier has been seen before; if not, the service opens an idempotent session keyed off the customer's identity plus the request identifier, and processes the request. The sentence that carries the whole implementation's weight: recording the token and all the changes made to service the request must happen together as one ACID unit, meaning a database guarantee that a set of changes either all take effect or none do. This has to be all-or-nothing because each half-failure breaks the contract in its own way. If the token is recorded but the resources aren't created, a later retry sees the token, assumes success, and replays a success that never actually happened. If the resources are created but the token isn't recorded, a later retry sees no token, assumes the request is new, and creates the resources a second time.
Then the post's most distinctive argument: what the replay should actually say. Returning ResourceAlreadyExists to a retry technically satisfies idempotency (it has no effect on the service), but the post rejects it because it has an effect on the client: a caller that never saw the resource get created now has to handle a message saying it already exists, and because that reply is different from a normal success, the caller's code has to branch to deal with it - which is exactly what makes retrying by default hard to offer. What AWS does instead is send back a response that means the same thing every time. For a set period, any repeat of the request with the same identifier gets a reply carrying the same meaning as the first success, even though the details may differ. That consistency is what lets the SDK and CLI generate a token when the caller doesn't supply one, reuse it across automatic retries, and keep application code completely unaware that any retry happened; the CLI worked example shows the same ClientToken echoed back, the replayed response similar but not identical, the instance now 'running' instead of 'pending.'
Two edge cases finish the contract. Late-arriving retries: a delayed retry lands after another actor has already terminated the instance, and EC2 honors the original contract anyway, replaying a same-meaning success whose body shows the state as 'terminated,' following the principle of least astonishment (do the thing that will surprise the caller least): a consistent, predictable reply is better than a technically more up-to-date one that breaks the pattern. That forces a retention policy: identifiers can't live forever, since a future request could collide with an ancient token, so EC2 keeps them for the resource's lifetime plus an interval after which a late retry would either have arrived or no longer be valid. And changed parameters under a reused token: the safest assumption is that the customer meant a different outcome, so the service returns a validation error naming the mismatched parameter, which means storing the original request's parameters alongside the token. The conclusion is candid about the bill: this contract costs real service-side complexity, and it is not right for every case; sometimes the strict at-most-once contract is worth the time, and sometimes shipping faster on a looser one serves customers better.
Tradeoffs
- The contract moves the complexity from every client to the one provider, and at platform scale that direction is the whole argument. The client gets one simplifying rule (any non-validation error, retry until success), which removes reconciliation logic and error-handling boilerplate from millions of callers; the service pays once, with sessions, stored parameters, a retention policy, and the atomicity requirement. The post's closing admits the bill is real and not always worth paying: building a strict at-most-once contract competes for the same engineering time as building features that set the product apart.
- Caller-provided tokens beat computed hashes because only the caller knows intent, but that also moves part of the guarantee into the caller's hands. A parameter hash wrongly merges the customer who actually wants two identical instances; a ClientToken never does, but now its safety depends on the client using it correctly: a new token for a new intent, the same token reused on a retry. AWS closes most of that gap by having the SDK and CLI generate and reuse tokens automatically, so the contract quietly protects users from the very thing it needs them to get right.
- The token and the changes have to commit as one all-or-nothing unit, and that single requirement is where the real difficulty lives. Record the token without the resources and a retry replays a success that never happened; create the resources without the token and a retry duplicates them. For a request whose work spans placement, volumes, and network interfaces, making all of that commit together is genuinely hard, and it is the one line in the post that costs the most to build.
- AWS replays a same-meaning response rather than the truthful-but-disruptive alternative. ResourceAlreadyExists has no effect on the service but changes the caller's flow of control, which counts as a side effect from the client's point of view, so AWS instead sends back a reply that means the same thing as the first success, which is what lets the SDK retry without the caller ever noticing. The price is that the service has to store enough of the first outcome to reconstruct a matching reply later, and deciding what still counts as 'the same meaning' is a judgment each service has to make once the real state has moved on (an instance goes from pending to running to terminated).
- Least astonishment means the replay keeps working even after the resource has been deleted. A late retry after termination still gets a success whose body says 'terminated,' consistent with every earlier response to that token, and surprising only to a caller who reads the status code instead of the state. The trade is deliberate, consistency of experience over freshness of the answer, and it puts an obligation on callers: read the current state from the response body rather than assuming it from the fact that the call succeeded.
- Token retention is a clock nobody sees, and the guarantee only lasts as long as the clock runs. Keep tokens too briefly and a genuinely late retry duplicates the resource; keep them forever and a future identifier can collide with an ancient one. EC2's answer (the resource's lifetime plus a reasonable-lateness interval) is a judgment made per resource type, which means 'at most once' across the platform quietly comes with a time limit that varies from service to service.
Patterns in this article
- Idempotency Keys
The caller attaches a unique identifier to a request, and the service treats any later request carrying the same identifier as the same request, so a retry replays the first outcome instead of doing the work twice. Stripe, Shopify, and Airbnb have all appeared in the library making this call from the client-facing side; AWS's piece is the platform provider's view of the same contract (EC2 calls the identifier a ClientToken). The reason a caller-supplied key beats a hash of the request is intent: identical parameters don't mean identical intent, and only the caller knows whether a second identical request is a mistaken duplicate or a genuine second order.
- Retryable Error Classification
AWS turns the whole client contract into a single line to classify errors by: any error that isn't a validation error can be retried until it succeeds. That one rule is what the SDK's default retry policy runs on - it retries everything on the non-validation side and stops on the validation side. And the validation side does real work here, not just rejection: when a reused token arrives with different parameters, the service returns a validation error precisely so the retry won't blindly repeat against a changed request. Airbnb drew the same line inside an internal framework with per-exception judgments; AWS draws it at the public API boundary itself, where the SDK can act on it for every caller.
- Atomic Phases
Recording the idempotency key and making all the changes that service the request have to commit as one all-or-nothing unit (an ACID transaction). The post is blunt about why: if the key is saved but the work fails, a retry replays a success that never happened; if the work commits but the key isn't saved, a retry does the work a second time. The guarantee that makes retries safe is not the key by itself but the key and the work landing together or not at all.
Also solving this
Other systems in behindscale's Ambiguous failure under retry class: