Problem · seen at 3 companies
Blast radius scales with cluster size
The bigger you build the shared cluster, the more a single dead node takes down with it.
The wall
In one large shared cluster the cost of a single failure grows with node count — a lone slow or dead node degrades a disproportionate share of operations, and coordination overhead worsens with size — so the fix is smaller isolated cells. 3 teams hit this wall; the breakdowns below are the evidence.
Same wall, 3 systems
Discord
Discord Engineering
One slow node failed ~40% of indexing batches: blast radius scaled with cluster size. Read the breakdown →
Amazon (AWS)
Amazon Builders' Library
A shared fleet makes every failure everyone's failure — one poison request reached the whole service. Read the breakdown →
Shopify
Shopify Engineering
Every new shard was a new way to take the whole platform down. Read the breakdown →