One Pod Online: Shopify's Cure for Platform-Wide Failure

The 2015 ceiling was vertical and absolute: it was no longer possible to continue buying a larger database server, so Shopify sharded MySQL and kept growing. The post is candid about the exchange rate — what we gained in performance and scalability we lost in resilience — and precise about the mechanism. Throughout the codebase lived a fan-out idiom: Sharding.with_each_shard do some_action end. Any platform-scoped action — a sweep, a migration-shaped task, anything that touched all shops — iterated the full shard set, and if any shard was down, that entire action was unavailable across the platform.

The arithmetic of that idiom is the class's arithmetic. An action spanning N shards is up only when all N are up; its availability is the product of theirs, and it decays with every shard added. Growth therefore degraded resilience by construction: the fleet that solved the capacity problem was simultaneously manufacturing new single points of platform failure, one per shard. Shopify's own operational history had already taught the sibling lesson about shared resources — a single Redis shared across all of Shopify once took the whole platform down, an incident remembered internally as Redismageddon, distilled to the rule: avoid any resource shared across all of Shopify. Between the shared-datastore path and the fan-out path, the platform's failure domains all had the same size — everything. In 2016, the team sat down to reorganize the runtime architecture around a harder requirement than capacity: a shard's failure must not be able to spiral into a platform outage at all.

The pod is the unit that requirement produced. A pod is a set of shops living on a fully isolated set of datastores — MySQL, and per the platform's later shape, the caches and queues beside it — so no datastore is shared across pods, and a hot or broken store can consume only its own pod's database time. Compute stays pooled for efficiency: job workers, app servers, and load balancers are shared resources. The discipline that keeps pooling from re-coupling fates is the architecture's first law: a shared resource may only ever communicate with a single pod at a time; no action reaches across pods. Isolation alone, the post notes, buys capacity and containment but doesn't kill the fan-out — that takes the second law: every unit of work, web request or delayed job, is assigned to exactly one pod. The consequence is the post's thesis in one line: serving a request only requires a single pod to be online. Adding a pod adds capacity without adding a new way for pre-existing pods to fail, because nothing ever waits on more than one.

Assignment needs an assigner. Sorting Hat lives in the load balancers and, using a list of rules, matches every incoming request to its pod, attaches a header, and forwards it to the right location — across multiple data centers. Application servers use that header to connect to the one datastore set that should answer, keeping shared compute capacity while querying exactly one pod per request. The router is where the isolation boundary is actually enforced; the pod map is a routing decision made once, at the edge, rather than a query-time negotiation.

The same atom reorganized disaster recovery. Shopify already ran a recovery data center; podding refined it from a platform-sized failover to a per-pod one. Each pod is assigned a pair of data centers — one active, one recovery — and Pod Mover relocates a pod to its recovery site in about a minute without dropping requests or jobs. That granularity upholds the core promise (a single pod's failure cannot spiral) and turns the scariest operation in infrastructure into a routine one: evacuating an entire data center is nothing more than evacuating its pods one at a time, and Shopify moves pods around daily — because manually moving services and ensuring correctness had quickly become untenable, and stressful. The platform's later shape testifies to where this went: over a hundred pods, and no major outage affecting all of Shopify since the architecture landed — an outage today touches a single pod or region.