Asked Twice, Done Once: How AWS Builds Idempotent APIs
An EC2 instance launch decomposes into calls for placement, EBS volumes, network interfaces, and VM provisioning — and when one call fails transiently, the simplest cure is the best one: retry until it succeeds, a pattern so effective AWS bakes default retries into its SDKs. Malcolm Featonby's Builders' Library piece is about the assumption that cure quietly rests on: that a retried call has no additional side effects. The motivating dilemma is a customer running a singleton workload whose RunInstances call times out — retrying could launch a second instance, with dire consequences. AWS's answer is the idempotent API contract from the provider's side: a caller-supplied unique client request identifier (EC2's ClientToken), an idempotent session recorded atomically with every mutation, semantically equivalent responses replayed for the token's lifetime — even, by the principle of least astonishment, after the resource has been deleted — and a validation error when the same token arrives with different parameters, because a changed request is a changed intent.
Run a singleton workload's RunInstances with the response cut mid-flight, and retry your way into two instances — then arm the ClientToken and watch the replay. Break the token's atomicity and duplicate anyway; try a synthetic hash and dedupe a customer's real intent; terminate the instance and meet the late retry that replays success about the dead.
Problem
The post opens with the pattern that makes retries load-bearing at AWS: complex operations decompose into a controlling process calling smaller services — an EC2 launch touches placement, EBS volume creation, elastic network interfaces, VM provisioning — and the controlling process must shepherd all of them into a known good state. A surprisingly large number of transient faults yield to simply retrying, which is why AWS SDKs retry by default on network I/O failures, server-side faults, and rate limiting: it deletes a whole category of undifferentiated boilerplate from every calling service. But the default rests on a simplifying assumption the post names precisely — that the operation can be retried without additional side effects. It would be undesirable for a retried EBS-volume call to produce two volumes.
The scenario that sharpens the assumption into a dilemma: a customer's provisioning process runs a singleton workload — at most one EC2 instance at any time — and its RunInstances call receives no response. Is the workload running? Simply retrying could produce multiple workloads, with dire consequences; not retrying strands the launch. The no-contract escape is reconciliation — query, compare, decide — which the post dismisses on two grounds: it is a lot of heavy lifting to compensate for an infrequent edge case, and it doesn't even resolve the uncertainty, because a discovered instance might have been created by a different provisioning process. Whether this instance is yours matters, and no amount of inspection after the fact can tell you.
Even the obvious dedup shortcut fails on intent. A synthetic token — a hash of the request parameters — treats identical-looking requests as duplicates, which is plausible for two near-simultaneous CreateTable calls and wrong for EC2: the customer may genuinely want two identical instances. Sameness of parameters is not sameness of intent, and only the caller knows the difference.
Solution
AWS's preferred contract makes the caller declare intent: a unique caller-provided client request identifier — the ClientToken in EC2's API — where requests from the same caller with the same identifier are duplicates by definition. The choice carries operational dividends the post enumerates: the identifier lands in CloudTrail logs, making intent auditable; the created resource is labeled with it (visible in DescribeInstances), so customers can trace any resource to the request that made it.
Server-side, receiving a request means checking whether the identifier has been seen; if not, the service creates an idempotent session keyed off the customer identifier plus the request identifier, and processes the request. The sentence that carries the implementation's whole weight: recording the idempotent token and all mutating operations related to servicing the request must together meet ACID properties — all or nothing. Either failure order breaks a different side of the promise: token recorded without resources, and the retry replays a success that never happened; resources created without the token, and the retry duplicates them.
Then the post's most distinctive argument: what the replay should say. Returning ResourceAlreadyExists to a retry technically meets idempotency — no side effect on the service — but the post rejects it for having a side effect on the client: a caller that never saw the resource created must now handle its existing, and the changed return code changes the caller's flow of execution, which makes retry-by-default hard to offer. The alternative AWS standardizes is the semantically equivalent response: for some interval, every response to the same identifier carries the same meaning as the first success. That property is what lets the SDK and CLI generate a token when the caller doesn't supply one, reuse it across automatic retries, and keep application code completely unaware retries happened — the CLI worked example shows the same ClientToken echoed back, the replayed response similar but not identical, the instance now 'running' instead of 'pending.'
Two edge cases finish the contract. Late-arriving retries: a delayed retry lands after another actor has terminated the instance — and EC2 honors the original contract anyway, replaying a semantically equivalent success whose body shows state 'terminated,' on the principle of least astonishment: the consistent experience beats the technically-fresher surprise. Which forces retention policy: identifiers can't live forever (a future request could collide with an ancient token), so EC2 retains them for the resource's lifetime plus an interval after which late retries would either have arrived or no longer be valid. And changed parameters under a reused token: safest to assume the customer intended a different outcome — return a validation error naming the parameter mismatch, which requires storing the original request's parameters alongside the token. The conclusion is candid about the bill: this contract costs real service-side complexity, and it is not right for all solutions — sometimes the rigorous at-most-once contract is worth the time, and sometimes innovating faster on a looser contract serves customers better.
Tradeoffs
- The contract relocates complexity from every client to one provider, and at platform scale that direction is the whole argument. The client's simplifying assumption — any non-validation error, retry until success — deletes reconciliation workflows and fault-handling boilerplate from millions of callers; the service pays once, with sessions, parameter storage, retention policy, and atomicity requirements. The post's closing concedes the bill is real and not always worth paying: a rigorous at-most-once contract competes for the same engineering time as differentiating features.
- Caller-provided tokens beat synthetic hashes because only the caller knows intent — and that moves part of the guarantee into the caller's hands. A parameter hash wrongly dedupes the customer who wants two identical instances; a ClientToken never can, but its safety now depends on client discipline: unique per intent, reused exactly on retry. AWS closes most of that hole by having the SDK and CLI generate and reuse tokens by default, which is the contract quietly protecting users from its own preconditions.
- The token and the mutations must commit as one ACID unit, and that sentence is where the difficulty actually lives. Token-without-resources replays a success that never happened; resources-without-token duplicates on retry. For a request whose servicing spans placement, volumes, and network interfaces, 'all or nothing' is a genuinely hard property — the single line in the post that costs the most to build.
- Semantic equivalence is chosen over the truthful-but-disruptive alternative. ResourceAlreadyExists has no service-side effect but changes the caller's execution flow — a side effect from the client's perspective — so AWS replays meaning-equivalent responses instead, which is precisely what makes SDK-invisible default retries possible. The price: the service must retain enough of the first outcome to reconstruct an equivalent response, and 'equivalent' is a per-service judgment when state legitimately advances (pending becomes running becomes terminated).
- Least astonishment extends the replay past the resource's death. A late retry after termination gets a success whose body says 'terminated' — consistent with every earlier response to that token, surprising only to a caller who reads the status code and not the state. The trade is deliberate: consistency of experience over freshness of verdict, and it obligates callers to read current state from the response body rather than inferring it from success.
- Token retention is a clock nobody sees, and the guarantee is only as long as the clock. Retain too briefly and a genuinely late retry duplicates the resource; retain forever and future identifiers can collide with ancient ones. EC2's answer — resource lifetime plus a reasonable-lateness interval — is a per-resource-type judgment, which means 'at most once' everywhere in the platform carries a quietly bounded time horizon that varies by service.
Patterns in this article
- Idempotency Keys
Fifth article on the pattern, completing four company perspectives with the provider's: Stripe writes the contract, Shopify motivates it at volume, Airbnb builds the server interior — and AWS shows why the key must come from the caller at all: a synthetic parameter hash cannot distinguish a duplicate from a customer who wants two identical instances. Distinctive provider-side additions: the token labels the created resource and lands in CloudTrail (intent made auditable), and a reused token with changed parameters is a validation error, because sameness of key with difference of parameters is a contradiction of intent.
- Retryable Error Classification
Second company, and the classification stated as the entire client contract: any error that isn't a validation error can be overcome by retrying until success. Airbnb's version is an internal framework discipline with per-exception judgments; AWS's is the public API boundary itself — the validation/non-validation line is what the SDK's default retry policy executes against, and the parameter-mismatch error shows the non-retryable side doing intent-protection work.
- Atomic Phases
Second company. Airbnb fences Pre-RPC and Post-RPC inside single enclosing transactions; AWS states the same law from the provider's seat in one sentence — recording the idempotent token and all mutating operations must together be ACID, all or nothing — and names both failure orders the atomicity forecloses: a token without its resources replays a phantom success, resources without their token duplicate on retry.
Also solving this
Other systems in behindscale's Ambiguous failure under retry class: