The catalog

Browse by problem class.

Grouped by the crux — the bottleneck that made each system hard. Filter by company to see who else hit the same wall.

Ambiguous failure under retry

5 systems

A request that times out or fails mid-flight leaves the caller unable to tell whether it already took effect, so a retry risks repeating an operation that in fact succeeded.

SEEN AT Airbnb · Amazon (AWS) · Segment · Shopify · Stripe

Buffer degrades under backlog

5 systems

A buffer that stops absorbing exactly when the backlog it exists for arrives — whether the buffer's substrate degrades under the backlog, or its access semantics let the backlog capture the buffer.

SEEN AT DoorDash · Meta · Segment · Slack · Uber

Single-cluster scaling ceiling

5 systems

A single database cluster becomes both a capacity ceiling and a shared-fate domain: it can only be scaled by ever-larger machines, and any incident on it degrades everything built on top — so relief means moving whole tables or domains onto separate clusters.

SEEN AT Airbnb · GitHub · GitLab · Google · Slack

Priority-blind load shedding

4 systems

Under overload the system drops work without regard to its importance, so shedding meant to protect the service also sacrifices its most critical traffic.

SEEN AT Amazon (AWS) · Netflix · Stripe · Uber

Single-table scaling ceiling

4 systems

A single table's size or write volume outgrows what any one database instance can serve, and because a table is the smallest unit vertical partitioning can move, only splitting the table itself across shards relieves it.

SEEN AT Canva · Figma · Notion · Pinterest

Blast radius scales with cluster size

3 systems

In one large shared cluster the cost of a single failure grows with node count — a lone slow or dead node degrades a disproportionate share of operations, and coordination overhead worsens with size — so the fix is smaller isolated cells.

SEEN AT Amazon (AWS) · Discord · Shopify

Gray failure defeats automatic detection

3 systems

When a failure is partial and inconsistent, different components hold contradictory views of what is healthy, so automated detection cannot converge on a fault the system can't agree exists — and mitigation must fall to a human acting on ambiguous evidence.

SEEN AT Cloudflare · Meta · Slack

Observer shares fate with observed

3 systems

A system depends on the very infrastructure it exists to watch or protect — most commonly monitoring running on the servers it monitors — so a failure in that foundation disables the safeguard exactly when it is needed.

SEEN AT Airbnb · Datadog · Roblox

Partial completion under crashes

3 systems

A multi-step process interrupted between steps leaves its outcome and its side effects undetermined — some work done, some not, no safe way to resume without either dropping or duplicating it — and, at scale, often no way even to enumerate which steps remain.

SEEN AT Airbnb · Netflix · Uber

Degraded state outlives its trigger

1 system

A failure that persists after its trigger is removed, because the degraded state regenerates itself; exits are external (shed load below the tipping point, or add capacity).

SEEN AT Slack

Mitigation scoped narrower than the failure

1 system

A failure propagates through interactions across components, but each defense measures and acts only within one component, so every mechanism can behave correctly while the failure spreads between them — and the containing move lies outside any single defense's reach. Boundary against retry-amplified overload: retry amplification is one specific transmission mechanism whose fix (budgets, jitter, backoff) is local to the retrying client; this class is about the scope mismatch between defense and failure, whatever the transmission mechanism.

SEEN AT DoorDash

Retry-amplified overload

1 system

When failures are caused by overload, clients retrying to improve their own odds spend the struggling dependency's scarce capacity, multiplying load at the worst moment — and naive backoff does not fix it, because clients that failed together retry in sync.

SEEN AT Amazon (AWS)

Unrecorded config outlives its authors

1 system

Load-bearing configuration that outlives its authors and its record, detonated by an environmental change nothing could check it against.

SEEN AT Reddit