Ten Bounds on Failure: Resilient Payment Systems at Shopify
At volume, one-in-a-million payment failures happen constantly — and a timeout can't tell success from failure.
Grouped by the crux — the bottleneck that made each system hard. Filter by company to see who else hit the same wall.
Showing 14 of 14 breakdowns across 10 problem classes.
A request that times out or fails mid-flight leaves the caller unable to tell whether it already took effect, so a retry risks repeating an operation that in fact succeeded.
At volume, one-in-a-million payment failures happen constantly — and a timeout can't tell success from failure.
A timed-out request leaves the client unable to tell whether it already happened.
A multi-step process interrupted between steps leaves its outcome and its side effects undetermined — some work done, some not, and no safe way to resume without either dropping or duplicating it.
A multi-step process crashing between steps leaves its outcome, and its side effects, to chance.
Every service rebuilds the same crash-durable machinery — the duplicated scaffolding is what doesn't scale.
Under overload the system drops work without regard to its importance, so shedding meant to protect the service also sacrifices its most critical traffic.
Priority-blind shedding spends the drop budget on rides to protect batch jobs.
One concurrency limiter shed traffic blind to priority, dropping critical playback with everything else.
A single table's size or write volume outgrows what any one database instance can serve, and because a table is the smallest unit vertical partitioning can move, only splitting the table itself across shards relieves it.
The biggest tables outgrew any single instance, and vertical partitioning can't split a table.
The block table's writes outgrew the instance — VACUUM stalled, transaction-ID wraparound loomed.
In one large shared cluster the cost of a single failure grows with node count — a lone slow or dead node degrades a disproportionate share of operations, and coordination overhead worsens with size — so the fix is smaller isolated cells.
One slow node failed ~40% of bulk operations — blast radius scaled with cluster size.
A component whose job is to absorb backlog is served by a substrate that slows down as the backlog grows, so the very condition it exists to handle erodes its ability to handle it.
The queue's own substrate slows down exactly as the backlog it must absorb grows.
When a failure is partial and inconsistent, different components hold contradictory views of what is healthy, so automated detection cannot converge on a fault the system can't agree exists — and mitigation must fall to a human acting on ambiguous evidence.
Gray failure in one availability zone defeated automatic detection — components disagreed on what was down.
The monitoring system shared its fate with the very infrastructure it watched.
When failures are caused by overload, clients retrying to improve their own odds spend the struggling dependency's scarce capacity, multiplying load at the worst moment — and naive backoff does not fix it, because clients that failed together retry in sync.
Retries amplify overload by spending the server's scarce capacity — and backoff alone can't stop it.
A single database cluster becomes both a capacity ceiling and a shared-fate domain: it can only be scaled by ever-larger machines, and any incident on it degrades everything built on top — so relief means moving whole tables or domains onto separate clusters.
A decade of code assumed one shared database — the real ceiling was the cluster.