Problem · seen at 5 companies
Ambiguous failure under retry
The request timed out, which tells you it didn't answer — not whether it actually happened.
The wall
A request that times out or fails mid-flight leaves the caller unable to tell whether it already took effect, so a retry risks repeating an operation that in fact succeeded. 5 teams hit this wall; the breakdowns below are the evidence.
Same wall, 5 systems
Shopify
Shopify Engineering
At volume, one-in-a-million payment failures happen constantly - and a timeout can't tell success from failure. Read the breakdown →
Amazon (AWS)
Amazon Builders' Library
A timed-out client can't tell if the work happened, and retrying risks doing it twice. Read the breakdown →
Airbnb
Airbnb Engineering
A payment call that times out may have moved money; retrying must never move it twice. Read the breakdown →
Segment
Segment Blog
Clients that miss the response re-send events the server has already processed. Read the breakdown →
Stripe
Stripe Engineering
A timed-out request leaves the client unable to tell whether it already happened. Read the breakdown →
Patterns in this class
Every breakdown
Shopify Engineering · Jul 28, 2022Ten Bounds on Failure: Resilient Payment Systems at ShopifyAmazon Builders' Library · Jan 15, 2021Asked Twice, Done Once: How AWS Builds Idempotent APIsAirbnb Engineering · Apr 16, 2019At Most Once: Orpheus and the Idempotent Payments Library at AirbnbSegment Blog · Jun 29, 2017Almost Exactly Once: Segment's Billion-Message Dedupe LedgerStripe Engineering · Feb 22, 2017Designing robust and predictable APIs with idempotency