One Twenty-Eighth: Shuffle Sharding and the Arithmetic of Blast Radius
Shuffle sharding is a way to limit how much of a service one bad customer can break, without buying extra hardware. This Builders' Library piece, by Colm MacCárthaigh, makes that blast radius something you configure. Start with eight worker machines where any machine can handle any request: efficient, but every customer shares all eight, so one customer's bad traffic can break everything. Splitting the eight into four fixed groups of two caps the damage at a quarter of the service, but everyone in the hit group goes fully down. Shuffle sharding instead gives each customer its own random pair of the eight machines. There are 28 possible pairs from eight machines, so one bad customer now affects only about 1/28th of the others, and because any two customers share at most one machine, a customer whose software just resends failed requests barely notices. Amazon's DNS service, Route 53, runs this at full scale: 2,048 machines, four to each customer domain (like example.com), giving 730 billion possible combinations.
Send one bad customer's traffic at the same eight machines three ways (all shared, four fixed groups, and shuffle sharding) and watch how much of the service goes down: 100%, then 25%, then about 1/28th. Then switch off the customers' automatic resends and see how much of the protection was really coming from their side.
Problem
Picture the standard setup: eight worker machines (servers, queues, databases, whatever your system is made of) sitting behind a load balancer, a piece that can send any incoming request to any of the eight. This design has real strengths before we find its weakness: it is efficient, and if one machine fails, the other seven pick up the work with little spare capacity needed.
That same flexibility is the weakness. Suppose one customer sends a poisonous request (one that makes a machine crash or hang), or simply floods the service with requests. The load balancer spreads that customer's traffic across all eight machines, so the problem hits every machine in turn. The result: every customer is hurt by one customer's problem, and adding more machines just gives the failure more places to reach, not fewer. For a service like DNS, where Amazon's Route 53 sits in front of some of the biggest sites on the internet, floods of junk traffic (called DDoS attacks) are a daily fact of life. There, 'one bad customer takes down everyone' is a fundamental danger, not a rare edge case.
Ordinary sharding is the first fix. Divide the eight machines into four groups of two, and assign each customer to one group. Now a poisonous customer can take down only their own group, so the damage is 25% of the service instead of 100%. But for the customers stuck in that group, the outage is total, and the only way to make the fraction smaller is to add more groups, which means more machines or smaller groups. So better isolation seems to require giving customers their own private machines, and private machines are exactly the cost that sharing machines across customers was meant to avoid.
Solution
Shuffle sharding keeps all eight machines shared and changes only how customers are matched to machines. Instead of fixed groups, each customer gets its own random pair of machines from the eight. Take two example customers, call them rainbow and rose. Rainbow is given machines 1 and 4; rose is given machine 1 and machine 8. Their pairs overlap on machine 1, and that overlap is the whole point, not a flaw.
Say rainbow's traffic turns poisonous. It takes down machines 1 and 4, so rainbow's own pair is down and the service loses a quarter of its capacity. But no other customer loses more than one machine, because any other pair overlaps rainbow's on at most one machine. Rose, for instance, loses machine 1 but still has machine 8. If rose's software can work around a lost machine (by resending the request, for example), rose keeps working with no interruption. That resend is doing quiet but essential work here: shuffle sharding removes half of a customer's capacity, and the resend hides that missing half.
Now the arithmetic that gives the pattern its name. From eight machines, there are 28 different ways to pick a pair (machine 1 with 2, 1 with 3, and so on, up to 28 in all). So with hundreds of customers spread across those 28 pairs, one bad customer affects only about 1/28th of them. That is seven times better than four fixed groups, from the very same eight machines. And unlike most improvements, this one gets better as you grow: more machines mean far more possible pairs. With enough machines there are more possible pairs than customers, so nearly every customer ends up alone on their own combination. Blast radius becomes a number you can configure, just by choosing how many machines there are and how many each customer gets.
Route 53 runs the full-scale version. Its capacity is arranged as 2,048 machines (called virtual name servers, virtual so they can be moved between physical servers as needs change), and each customer domain (like example.com) is given a set of four of them. That makes a staggering 730 billion possible sets: enough for every domain to get a unique four, and to guarantee that no two domains ever share more than two machines. When one domain is hit by a DDoS attack, shuffle sharding lets Route 53 spot the targeted customer and move it onto special dedicated attack-handling capacity, alongside Amazon's own Shield service that filters out attack traffic. Everyone else keeps working normally. Amazon has since used the pattern across many of its systems, extended it to several layers at once (so a customer's own customers can be isolated too), and released it as the open-source Route 53 Infima library. The quiet punchline: it is just a smarter way to arrange machines you already have, and it usually costs nothing extra.
Tradeoffs
- The protection is about averages, not a guarantee for every customer. A shuffle-sharded customer still loses their whole pair when their own traffic is the poison. And '1/28th' is an average across all customers, not a hard limit: the customers whose pairs happen to overlap the broken machines do feel real slowdowns, and they only stay up if their software can cope with a lost machine. Shuffle sharding makes the damage smaller; it does not give each customer the total protection that fully private machines would.
- The recovery depends on code you don't control. The 'no interruption' promise only holds for customers whose software resends failed requests; a customer whose software gives up on the first failure still sees the partial outage shuffle sharding was meant to prevent. So the pattern quietly leans on every customer's own code, much like Meta's and Slack's queues leaning on their consumers to handle repeats safely. And it can backfire if those resends pour in with no limits or pacing, piling more load onto an already-struggling service.
- Overlap buys the huge number of combinations, but it costs clarity. With fixed groups you can point at exactly who shares a group with whom. With shuffle sharding, who-shares-with-whom becomes a matter of probability, capped only in pairs (Route 53 promises no two domains share more than two machines). That is stronger protection overall, but harder to reason about for any single incident: to answer 'which customers does this one slow machine affect?' you now need the assignment list, not a simple picture.
- Shuffle sharding needs routing that knows every customer's assignment. It only works if the layer directing requests can reliably map each customer to their own machines, and keep that mapping steady even as machines are swapped out and capacity moves around. Route 53's machines are called 'virtual' for exactly this reason: the capacity underneath can move while each customer's assignment stays put. That is real logic a plain load balancer never needed, and now it is a piece you have to build and keep correct. Amazon's answer was to release it as the Infima library.
- The economics are the whole argument, and they point one way. Shuffle sharding rearranges machines you already have, usually at no extra cost, which is why a tight budget can actually push you toward it. But 'no extra cost' is true only for the hardware. The assignment logic, the per-customer routing, the monitoring that now has to think in combinations, and the dependence on customers' resend behavior are all real engineering effort. Cheap in hardware does not mean cheap in design.
- A related system, Discord's message search, shows where this pattern stops fitting. Discord's damage spread because each indexing batch fanned out across one big cluster, tying every operation to every machine. Their fix was many small physical clusters that keep each customer's data together, because a search has to run where its data lives. Shuffle sharding fits work where any machine can serve any request (DNS, queues, stateless services). When the data ties a customer's work to specific machines, the groups have to be real, separate machines, not overlapping virtual pairs. Same kind of problem, two different shapes of solution, and which shape you can use depends on where your data lives.
Patterns in this article
- Shuffle Sharding
This article is the original explanation of the pattern, written by the team that invented it. Each customer gets a virtual shard: a random combination of machines (2 of 8 in the small example, 4 of 2,048 in Route 53). One problem can reach at most 1 divided by the number of possible combinations, and because any two customers' sets share at most one machine, customers whose software resends failed requests keep working. Amazon extended it to several layers (recursive shuffle sharding) and released it as the open-source Route 53 Infima library.
- Fault Isolation
The article spells out the arithmetic: no sharding means a 100% blast radius, four fixed groups means 25%, and shuffle sharding means 1/28th, all from the same eight machines. Route 53's promise that no two domains share more than two of their four machines is fault isolation written as a rule about combinations, not a physical wall.
Also solving this
Other systems in behindscale's Blast radius scales with cluster size class: