Committed Nowhere: Reddit's Pi-Day Outage
Reddit was down for 314 minutes on Pi Day 2023 — March 14 — and the deepest cause was configuration that existed nowhere: set up years earlier by a team that no longer existed, hand-edited through a vendor's command-line tool that bypasses the normal Kubernetes API, committed to no repository, and keyed to a node label that Kubernetes 1.24 silently removed from running clusters. Two minutes after an engineer started a routine upgrade on the cluster running Old Reddit — the most critical legacy piece of the company's dependency graph — the site halted, and the responders were nearly blind: the cluster's metrics were Kubernetes-native and died with it, while the logs survived only because they were built low-level. Kubernetes has no downgrade path, so rolling back meant the thing everyone fears: restoring the cluster from backup, using a runbook written years earlier for a different container runtime, never once used against a production cluster, rewritten live as it was executed — and hiding one more trap, a backup that could be taken on any control-plane node but restored only on the same one. The postmortem's verdict names the real culprit above the label bug: inconsistency — a fleet of bespoke, hand-reared clusters — and the remedy is a discipline: standardize, and codify everything, so that no load-bearing state ever again exists without a record.
Run the incident from the operator's seat: watch every metric die at T+2 minutes, burn through four fix-forward attempts that each change nothing, make the restore call against a runbook written for a runtime you no longer run, walk into the same-node trap, then bring traffic back — jump to 100% and learn what cold caches do, or walk the eight steps home. Then read the two-second log clue and the YAML that took Reddit down.
Problem
The cluster in question ran the Legacy stack — Old Reddit — and the post is candid about the fleet it belonged to: Reddit adopted Kubernetes early, accumulating bespoke clusters built by hand with the kubeadm tool rather than any standard template, some too large for cloud-managed offerings to even support. Pets, not cattle, in the team's own words — with inconsistent upgrade cadence and configuration split between clusters. The Compute team's upgrade process was genuinely careful: versions tested against dedicated clusters, then rolled from lowest criticality to highest. The team had literally just finished the postmortem for a previous upgrade of this same cluster that had gone mildly wrong, for a fully resolved cause. The engineer kicked off the 1.23-to-1.24 upgrade just after 19:00 UTC. Everything seemed fine for about two minutes. Then the site came to a screeching halt.
Responders were on a call within three minutes and nearly blind: the cluster had completely lost all metrics (the edge CDN's stats survived by being intentionally separate). DNS gave the one strange clue — names in Consul, the service Reddit runs for cross-environment dynamic DNS, and in-cluster names wouldn't resolve, while public DNS worked fine. Pods took extraordinarily long to start and stop; sub-100MB container images took minutes to pull over multi-gigabit links; control-plane logs flowed heavily with no obvious error. Calico — the cluster's networking layer, which gives every pod its network interface and routes — was unhealthy, and the fix-forward attempts read like a checklist of reasonable moves that changed nothing: a stuck pod matching a known low-severity container-runtime bug (deleted; not the problem), the Calico aggregation proxy's pods deleted and simply never recreated, a rolling restart of the control plane (no change), the full off-and-on-again (no change), and — one genuine finding — write operations timing out not on the writes but on calls to the cluster's admission webhooks, so Open Policy Agent's webhook configurations were deleted and the timeouts vanished instantly. The cluster stayed dead.
Rolling back was Plan A from the first minutes, and the post delivers the industry-wide bad news plainly: Kubernetes has no supported downgrade procedure — upgrades run schema and data migrations automatically, and there is no reverse path. A downgrade is a restore from backup. The backup existed, taken as standard. The restore was another matter entirely.
Solution
Two hours in, out of constructive ideas, they made the hard call. Worker nodes would be invalidated by the restore anyway, so all of them were terminated first — twenty minutes of API calls on Reddit's largest cluster — and then a restore procedure began that nobody on the call had ever executed, against the company's favorite single point of failure. It had been written against a now end-of-life Kubernetes version, before the switch to the CRI-O container runtime, so its Docker-era commands had to be rewritten live as the team executed it. And it hid a trap the incident walked straight into: the backup procedure was written to run on any control-plane node, but the restore procedure assumed the same node — a mismatch nobody had ever hit, because the restore had only ever been rehearsed on pilot clusters, never production. They restored to the procedure's designated 'first' node, brought it up as a single-node control plane, saw the autoscaler spring to life (proof networking was back — then deliberately shut it off to regain control), hit AWS capacity exhaustion for the control-plane instance type, and then hit the trap: new control-plane nodes stuck joining, no error, unable to reach etcd — the cluster's consensus store — on the restored node, because the TLS certificates it presented carried the wrong hostname. A breakout group found it in minutes; certificates were regenerated with some fumbling and no documentation; the control plane was whole again.
Bringing traffic back was its own discipline, because Reddit's caches are load-bearing: scale to 100% immediately and thundering herds hit downstream services that had idled and scaled down during the outage. They started at 1% and walked up — 5, 10, 20, 35, 55, 80, 100 — with particularly touchy legacy services stopped manually and re-enabled with care. The outage was over. The cause was still unknown.
The investigation is the post's best chapter. Metrics died with the cluster, but the logs — low-level, deliberately not Kubernetes-native — survived: 3.9 billion log lines. The API server's logging exploded 5x at 19:04:49; its only hint was the already-known OPA timeouts. OPA's own logs stopped five seconds before that — a dead end. Calico's logs held the real clue: two seconds before the chaos, calico-node across the cluster dropped routes to the first upgraded control-plane node — expected — and then all routes for all nodes. Route reflectors: in large clusters, instead of every node peering with every node, a few designated nodes relay routes for everyone (recommended past 100 nodes; 'we add zero(s)'). Reddit's reflectors were the control-plane nodes, selected by the label node-role.kubernetes.io/master — the label Kubernetes 1.24 removed from running clusters. The selectors matched nothing; the mesh lost its relays; networking ceased. And the reason nobody could see it coming: the configuration was set up years ago by a departed team, lives in Calico-specific data managed only by hand through Calico's CLI, and was committed nowhere. One engineer happened to remember the feature existed, and proved the mechanism during the postmortem.
The remediation section refuses to stop at the label. The actual cause is named as inconsistency — nearly every critical cluster bespoke in some way, 'a natural consequence of organic growth, and one which has caused more outages than we can easily track.' The cure is a program: standardize and share components instead of bespoke configuration, build more pre-production clusters that can be tested confidently, tool the lifecycle of whole clusters so they are recreatable and alike, codify everything that can be codified — 'both to ensure consistent application and to have a clear historical record of the choices that we've made' — document in detail what can't be, and treat every remaining exception as a candidate for replacement.
Tradeoffs
- Load-bearing state that exists outside any record is a time bomb with no visible clock. The route-reflector configuration was correct for years — and unfindable: no repository, no breadcrumbs, its authors gone. Nothing in a code review, an upgrade checklist, or a changelog read could surface a dependency that was written down nowhere. The audit this class demands runs on a hard question: what does this system depend on that exists only inside the running system?
- Vendor-CLI-only configuration is a codification gap someone must close deliberately. Calico's route-reflector config expects fetch-by-CLI, hand-edit, re-upload — actively resisting management as code — and the honest options are exactly three: build the custom tooling that brings it into the record, replace the mechanism with one that can be recorded, or accept it as a documented, named exception. Reddit had a fourth state — unrecorded and forgotten — which is the one this outage retired.
- Upgrades without a reverse gear change the economics of every decision downstream. Kubernetes' automatic schema migrations mean no supported downgrade — rollback is a backup restore. That single fact stretched the incident: hours of fix-forward attempts were rational precisely because the fallback was feared, unrehearsed against production, and of unknown duration. Systems without cheap reversal make their operators gamble on forward fixes.
- A backup you've never restored in anger is a hope, not a capability — and asymmetric procedures are its sharpest edge. The runbook predated the container runtime in use, was rewritten live mid-incident, and hid the any-node-backup/same-node-restore mismatch that cost a round of TLS failures with no error message. Restore paths need the same rehearsal discipline as forward paths, on the real environment's shape, before the day they're the only option left.
- The modernization boundary held, and drew the real blast-radius lesson. Reddit's modern service API layers stayed up throughout; the outage reached most user flows anyway because the legacy cluster remains the most critical node in the dependency graph. Isolation work is only as good as what still depends on the thing you isolated — the post's own conclusion is to keep shrinking that dependency, not to celebrate the layers that survived.
- Recovery traffic is a workload with its own physics. Idle services scale down; caches go cold; 100% of normal demand aimed at a cold system is a thundering herd, not a return to normal. The 1-to-100 walk in eight steps, with touchy legacy services hand-gated, treats readmission as a staged rollout of load — the same discipline as any migration, applied in the direction nobody practices.
Patterns in this article
- No Uncommitted Config
Minted from the remediation's central vow — codify everything, 'both to ensure consistent application and to have a clear historical record.' The pattern: no load-bearing state may exist outside version control; state manageable only through a vendor CLI gets custom tooling to bring it into the record, gets replaced by a recordable mechanism, or becomes a named, documented exception — never a fourth thing that is simply unrecorded. The record's second job is archaeology: breadcrumbs for responders who arrive after the authors have left.
- Rehearsed Restore
Minted from the recovery's hardest lesson: a standard backup paired with a restore runbook written for an end-of-life Kubernetes and a retired container runtime, never executed against production, rewritten live during the outage — and hiding the any-node/same-node asymmetry that stalled the control plane on TLS mismatches with no error. Restore is a capability only if it is rehearsed on the production environment's real shape, kept current with every environmental change, and symmetric with the backup that feeds it.
- Universal Staged Rollout
Recurs in the direction nobody practices: readmission. Traffic returned at 1% and walked up through eight steps to 100%, with idle-scaled services and cold caches treated as what they are — a system that cannot absorb normal load yet — and touchy legacy services hand-gated back in. Staged rollout applied to recovery load rather than to a migration.
- Load-Bearing Cache
Second company for the round-28 mint, one round later: the readmission ramp exists because Reddit's caches are load-bearing — 'Reddit relies on a lot of caches to operate semi-efficiently,' and full traffic against cold caches produces thundering herds in downstream services that idled during the outage. (Conditional per the mint's merge-or-discard status; agent aligns.)