One Pod Online: Shopify's Cure for Platform-Wide Failure

Problem

The 2015 limit was hard and simple: there was no bigger database server left to buy, so Shopify split the database across many servers (sharding) and kept growing. What Shopify gained in performance and scale, it lost in resilience. The problem was a one-line shared action found all across the codebase: Sharding.with_each_shard do some_action end. Whenever a platform-wide task ran (a cleanup sweep, a bulk update, anything touching all shops), this command made it work through every shard in turn. So if any one shard was down, the task could not finish, and it failed for the entire platform.

WHY EVERY SHARD MADE IT WORSE

The math behind this is simple, and it works against you. A task that spans N shards works only when all N are up, so its chance of working is all their uptimes multiplied together, and that number drops with every shard added. So growth made things less reliable by design: the same servers that solved the capacity problem were quietly creating new ways to take the whole platform down, one per shard. Shopify had already learned a related lesson the hard way. A single Redis store shared across all of Shopify once took the whole platform down, an incident they call Redismageddon, which left a rule: avoid any resource shared across all of Shopify. Both problems, the shared store and the fan-out, had the same blast radius: everything. So in 2016 the team set out to reorganize around a harder goal than capacity: one shard's failure must not be able to grow into a platform-wide outage at all.

Solution

To meet that goal, Shopify built a new unit called the pod. A pod is a set of shops living on their own separate databases (MySQL, and later the caches and queues too). No database is shared between pods, so one shop's heavy or broken store can only slow down its own pod. Workers and servers stay shared for efficiency, and two rules stop that sharing from tying the pods back together:

• a shared worker or server may talk to only one pod at a time, and no task reaches across pods
• every unit of work, each web request and each background job, is assigned to exactly one pod

Isolation alone buys capacity and containment but does not remove the fan-out; assigning every unit of work to one pod is what does. The result is the whole idea in one line: serving a request needs just one pod online. Adding a pod then adds capacity without adding a new way for the existing pods to fail, because nothing ever waits on more than one. The isolation also protects capacity: since each pod's databases are its own, one busy shop can use up only its own pod's database time, never time that every other shop depends on.

EACH REQUEST TOUCHES EXACTLY ONE POD

Something has to do that assigning. A piece of software called Sorting Hat runs in the load balancers and uses a list of rules to match each incoming request to its pod. It tags the request with a header and forwards it to the right place, even across data centers. The application servers read that header and connect only to the one set of databases that should answer, so compute stays shared while each request touches just one pod. The router is where the isolation is actually enforced: which pod handles a request is decided once, at the entry point, and not worked out again for every query.

Pods also helped with disaster recovery. Shopify already ran a backup data center; pods turned one big platform-wide failover into many small ones, each covering just a single pod. Each pod is given a pair of data centers, one active and one backup, and Pod Mover moves a pod to its backup in about a minute without dropping requests or jobs. That small granularity keeps the core promise (one pod's failure cannot spread) and turns the scariest operation in infrastructure into a routine one. Evacuating a whole data center is now just moving its pods out one at a time, and Shopify does this daily, because doing it by hand had quickly become too error-prone and too stressful. The results show where this led: over a hundred pods, and no outage affecting all of Shopify since the architecture landed. An outage today touches a single pod or region.

DISASTER RECOVERY, POD BY POD

Tradeoffs

• Sharding solves capacity, but it quietly turns the reliability math against you. One database is one thing that can fail; N shards under a fan-out action are N things that must all be up at once, so platform reliability drops as their uptimes multiply together. Because it was just one short line, the with_each_shard command looked harmless wherever it appeared, so it was easy to miss that it tied every shard's fate together.
• Full isolation gives up exactly the queries a shared database made easy. No action may reach across pods, so anything platform-wide (search across all shops, combined analytics, admin sweeps) has to be rebuilt as slower background jobs that run outside the live request, often against pre-combined copies of the data. The architecture does not make those tasks easier; it makes them explicit and pushes them off the path that has to be fast.
• The wall between pods is made of rules, not hardware. The databases are physically separate, but the workers and app servers are shared, so what keeps one pod's work from reaching another is careful routing and connection handling, not a physical barrier. One code path that loops over pods, or one connection opened to the wrong pod, quietly re-connects the pods that the separate databases worked to keep apart, so one pod's failure can reach another again.
• Assigning each unit of work to one pod changes the shape of failure: instead of everyone slowing down a little, one pod's shops go fully down while every other shop is untouched. The architecture chooses WHO fails instead of HOW MUCH everyone fails. That is the right trade for a platform serving many merchants, and it is still a trade, and the full cost of it lands on whoever happens to be on the pod that went down.
• Per-pod disaster recovery lets the team practice moving a pod at any time, but it doubles the number of data centers you have to run. Pod-pair data centers plus Pod Mover turn a data-center failover from a rare, heroic event into a one-minute routine done daily. The price is backup capacity kept ready for every pod, plus mover tooling that has to relocate live work without dropping a job.
• The router is the one part the architecture cannot split into pods. Sorting Hat's rules ARE the isolation boundary: a wrong route is not a slowdown but a correctness bug (a request reaching the wrong pod's data), and the routing layer stays shared, so its own faults can reach the whole platform. This kind of architecture does not remove the shared, global parts; it shrinks them to the thinnest possible layer and keeps that layer as simple and boring as it can.
• This is one of three answers to the same problem, and the most absolute. When a growing fleet means one member's failure can reach everything, AWS's shuffle sharding makes the shared damage smaller by giving each customer a random combination of machines. Discord caps it by running many small clusters instead of one big one. Shopify goes furthest: rather than shrinking shared failure, it removes shared actions altogether, so no operation can span pods in the first place. The trade for that is rigidity: cross-pod work is not slow, it is simply not allowed, and everything platform-wide has to be rebuilt around that rule.

Patterns in this article

• Cell Architecture

  Shopify's pods are cell architecture in its plainest form: shops split into pods, each a fully isolated slice of the platform, with every unit of work assigned to exactly one pod so serving a request needs just one pod online. The design goes a step further than draining traffic away from a sick cell: it makes cross-pod work impossible in the first place, so a pod's failure has nowhere to spread because nothing spans pods.

• Generic Mitigation

  Pod Mover is a generic mitigation: one big lever, built and ready before anything goes wrong, that evacuates a pod to its backup data center in a minute without dropping requests or jobs. It works whatever has gone wrong with the pod's current data center. Because Shopify moves pods every day, pulling this lever is routine rather than something figured out in a panic mid-outage, which is what makes it dependable.

• Fault Isolation

  Shopify pushes fault isolation to its structural extreme: not making the blast radius smaller but removing it, with unshared databases, single-pod actions, and per-pod disaster recovery. The one part it cannot isolate is the router, so it concentrates all remaining shared, global behavior into that single thin layer, and keeps that layer as simple and boring as possible.

Also solving this

Other systems in behindscale's Blast radius scales with cluster size class:

• How Discord Indexes Trillions of MessagesDiscord
• One Twenty-Eighth: Shuffle Sharding and the Arithmetic of Blast RadiusAmazon (AWS)