Problems

Browse by problem class.

Grouped by the crux — the bottleneck that made each system hard. Filter by company to see who else hit the same wall.

A request that times out or fails mid-flight leaves the caller unable to tell whether it already took effect, so a retry risks repeating an operation that in fact succeeded.

SEEN AT Airbnb · Amazon (AWS) · Segment · Shopify · Stripe

A buffer that stops absorbing exactly when the backlog it exists for arrives — whether the buffer's substrate degrades under the backlog, or its access semantics let the backlog capture the buffer.

SEEN AT DoorDash · Meta · Segment · Slack · Uber

Under overload the system drops work without regard to its importance, so shedding meant to protect the service also sacrifices its most critical traffic.

SEEN AT Amazon (AWS) · LinkedIn · Netflix · Stripe · Uber

A single database cluster becomes both a capacity ceiling and a shared-fate domain: it can only be scaled by ever-larger machines, and any incident on it degrades everything built on top — so relief means moving whole tables or domains onto separate clusters.

SEEN AT Airbnb · GitHub · GitLab · Google · Slack

A single table's size or write volume outgrows what any one database instance can serve, and because a table is the smallest unit vertical partitioning can move, only splitting the table itself across shards relieves it.

SEEN AT Canva · Figma · Notion · Pinterest

In one large shared cluster the cost of a single failure grows with node count — a lone slow or dead node degrades a disproportionate share of operations, and coordination overhead worsens with size — so the fix is smaller isolated cells.

SEEN AT Amazon (AWS) · Discord · Shopify

When a failure is partial and inconsistent, different components hold contradictory views of what is healthy, so automated detection cannot converge on a fault the system can't agree exists — and mitigation must fall to a human acting on ambiguous evidence.

SEEN AT Cloudflare · Meta · Slack

A system depends on the very infrastructure it exists to watch or protect — most commonly monitoring running on the servers it monitors — so a failure in that foundation disables the safeguard exactly when it is needed.

SEEN AT Airbnb · Datadog · Roblox

A multi-step process interrupted between steps leaves its outcome and its side effects undetermined — some work done, some not, no safe way to resume without either dropping or duplicating it — and, at scale, often no way even to enumerate which steps remain.

SEEN AT Airbnb · Netflix · Uber

A failure that persists after its trigger is removed, because the degraded state regenerates itself; exits are external (shed load below the tipping point, or add capacity).

SEEN AT Slack

A failure propagates through interactions across components, but each defense measures and acts only within one component, so every mechanism can behave correctly while the failure spreads between them — and the containing move lies outside any single defense's reach. Boundary against retry-amplified overload: retry amplification is one specific transmission mechanism whose fix (budgets, jitter, backoff) is local to the retrying client; this class is about the scope mismatch between defense and failure, whatever the transmission mechanism.

SEEN AT DoorDash

A placement decision (device, tier, region, replication level) must be made before the access pattern that would justify it exists — only files/objects whose placement decision precedes its evidence belong to this class. Systems where prediction merely optimizes an answer to some other wall (an ML-tuned cache atop a solved placement, a forecast feeding an autoscaler) do NOT belong: there prediction is the answer, not the wall.

SEEN AT Google

When failures are caused by overload, clients retrying to improve their own odds spend the struggling dependency's scarce capacity, multiplying load at the worst moment — and naive backoff does not fix it, because clients that failed together retry in sync.

SEEN AT Amazon (AWS)

Load-bearing configuration that outlives its authors and its record, detonated by an environmental change nothing could check it against.

SEEN AT Reddit