Ten Bounds on Failure: Resilient Payment Systems at Shopify
At volume, one-in-a-million payment failures happen constantly — and a timeout can't tell success from failure.
Grouped by the crux — the bottleneck that made each system hard. Filter by company to see who else hit the same wall.
Showing 25 of 25 breakdowns across 11 problem classes.
A request that times out or fails mid-flight leaves the caller unable to tell whether it already took effect, so a retry risks repeating an operation that in fact succeeded.
At volume, one-in-a-million payment failures happen constantly — and a timeout can't tell success from failure.
A payment call that times out may have moved money; retrying must never move it twice.
A timed-out request leaves the client unable to tell whether it already happened.
A component whose job is to absorb backlog is served by a substrate that slows down as the backlog grows, so the very condition it exists to handle erodes its ability to handle it.
The queue's own substrate slows down exactly as the backlog it must absorb grows.
Hundreds of destinations fail daily; a shared queue makes every failure everyone's backlog.
A full Redis queue couldn't dequeue — draining required the very memory the backlog consumed.
A multi-step process interrupted between steps leaves its outcome and its side effects undetermined — some work done, some not, no safe way to resume without either dropping or duplicating it — and, at scale, often no way even to enumerate which steps remain.
A multi-step process crashing between steps leaves its outcome, and its side effects, to chance.
Every service rebuilds the same crash-durable machinery — the duplicated scaffolding is what doesn't scale.
Half-done multi-day processes lived inside pub/sub code; nobody could answer what remained.
Under overload the system drops work without regard to its importance, so shedding meant to protect the service also sacrifices its most critical traffic.
Priority-blind shedding spends the drop budget on rides to protect batch jobs.
One concurrency limiter shed traffic blind to priority, dropping critical playback with everything else.
Under overload, undifferentiated dropping would spend charge creation to keep test traffic alive.
A single table's size or write volume outgrows what any one database instance can serve, and because a table is the smallest unit vertical partitioning can move, only splitting the table itself across shards relieves it.
The biggest tables outgrew any single instance, and vertical partitioning can't split a table.
The block table's writes outgrew the instance — VACUUM stalled, transaction-ID wraparound loomed.
The pins table outgrew any one machine; every clustered alternative broke first.
In one large shared cluster the cost of a single failure grows with node count — a lone slow or dead node degrades a disproportionate share of operations, and coordination overhead worsens with size — so the fix is smaller isolated cells.
One slow node failed ~40% of bulk operations — blast radius scaled with cluster size.
A shared fleet makes every failure everyone's failure — one poison request reached the whole service.
When a failure is partial and inconsistent, different components hold contradictory views of what is healthy, so automated detection cannot converge on a fault the system can't agree exists — and mitigation must fall to a human acting on ambiguous evidence.
Gray failure in one availability zone defeated automatic detection — components disagreed on what was down.
A switch failed partially instead of fully, and every failover layer built for clean up-or-down deaths misfired.
The monitoring system shared its fate with the very infrastructure it watched.
Telemetry depended on the failing Consul cluster — responders diagnosed a 73-hour outage nearly blind.
A single database cluster becomes both a capacity ceiling and a shared-fate domain: it can only be scaled by ever-larger machines, and any incident on it degrades everything built on top — so relief means moving whole tables or domains onto separate clusters.
A decade of code assumed one shared database — the real ceiling was the cluster.
One monolithic database carried a third of all writes — growth would overwhelm it within months.
A failure propagates through interactions across components, but each defense measures and acts only within one component, so every mechanism can behave correctly while the failure spreads between them — and the containing move lies outside any single defense's reach. Boundary against retry-amplified overload: retry amplification is one specific transmission mechanism whose fix (budgets, jitter, backoff) is local to the retrying client; this class is about the scope mismatch between defense and failure, whatever the transmission mechanism.
Failures spread through service interactions; every defense could see and act on one service.
When failures are caused by overload, clients retrying to improve their own odds spend the struggling dependency's scarce capacity, multiplying load at the worst moment — and naive backoff does not fix it, because clients that failed together retry in sync.
Retries amplify overload by spending the server's scarce capacity — and backoff alone can't stop it.