Stopping the Cause Didn't Stop the Outage: Slack's 2-22-22
On the morning of February 22, 2022, many users couldn't connect to Slack — including the engineer running the incident — and the detail that makes Laura Nolan's postmortem a classic arrives mid-story: responders paused the maintenance that had triggered everything, and it made no difference, because the failure no longer needed its trigger. The trigger was routine — a rolling upgrade of Consul, the service-discovery system that tracks which machines are alive. Each restarting agent briefly dropped its machine from the catalog, and Mcrib, the new control plane that assigns cache servers, replaced each 'departed' cache node with an empty spare — working faster and more correctly than its predecessor ever had. Two identical 25% upgrade steps had passed the week before without incident. The third landed at peak traffic, enough of the cache emptied, and a hidden amplifier took over: one common query, listing who's in your group direct messages, was stored in a way that made every single cache miss query every shard of the database. Load on the database grew faster than the miss rate itself, queries timed out, and the timeouts kept the cache from refilling — a loop that fed itself. Recovery came from operator intervention: throttling client boots (protecting already-connected users first), raising the limit in small steps after one too-large step caused a relapse, and rewriting the query to fetch only what was missing, from replicas.
Roll the Consul upgrade in 25% steps and watch two of them get absorbed — then land the third at peak traffic and cross the tipping point. Pause the rollout and learn the class's defining fact: it changes nothing. Throttle boots below the line, relax too fast and relapse, fix the scatter amplifier, and walk the system home in small steps.
Problem
When a Slack client starts a new session, it boots: fetching channel lists, preferences, and recent conversations before anything works. Booting leans on two tiers — Vitess, the MySQL-based datastore layer Slack migrated to (the same architecture dissected in this library's Vitess article), and a Memcached cache fleet in front of it, with Mcrouter distributing cache requests across servers and Mcrib, a newer control plane, deciding which servers are in the cache ring at all. Mcrib watches Consul, the service-discovery catalog, and when a cache node drops out, promotes an empty spare in its place; nodes that leave and rejoin get flushed first to avoid serving stale data. Its designers knew empty cache slots are dangerous — the stated design goal was a stable ring, flushing only when necessary — and Mcrib was genuinely better than the lock-based scheme it replaced.
The week's maintenance was a Consul agent upgrade rolling across the fleet in 25% steps, restarts deliberately slow and sequential to avoid destabilizing anything. A restarting agent deregisters its machine from the catalog and re-registers afterward — so from Mcrib's point of view, cache nodes kept briefly 'dying,' and Mcrib kept efficiently replacing them with empty ones. Two 25% steps had completed the prior week without incident. The third ran on February 22 and crossed a tipping point as the day hit peak traffic: enough of the cache was cold at exactly the moment demand was highest.
The amplifier was waiting in the data layout. The boot process asks which users share your group direct messages, and that data lived in a table sharded by user — efficient for finding a user's channels, structurally wrong for finding a channel's users. Vitess handles such queries by scattering them to every shard in the keyspace. Normally this was invisible: the answers are immutable, cached for a long time, almost never fetched from the database. But with the cache partially empty, nearly every booting user drove a query to every shard — and database load rose superlinearly as the hit rate fell, until most of those queries timed out. The timeouts were the trap closing: a query that times out fills no cache, and an unfilled cache guarantees the next miss.
Solution
The response ran on two tracks — understand the load, and reduce it — and the reduction is a study in shedding under pressure. The team throttled client boot operations specifically, knowing exactly what that meant: users not yet connected would stay out, so that users already connected would keep working, and the reduced query load would let the cache start filling. The first restrictive throttle worked. The first attempt to relax it taught the class's other lesson: they raised the limit too much, database load blew past sustainable limits again, failures climbed, and they had to drop back down and climb in smaller increments — because near a tipping point, the safe path back to full traffic is measured steps with the goodput watched at every one.
The amplifier got fixed in place. The scatter query was modified to read from the database only the entries actually missing from cache, rather than re-fetching everything on any miss; and because group-membership data is immutable — staleness is impossible — it was also allowed to read from Vitess replicas instead of only primaries, multiplying the capacity available to serve it. With the amplifier weakened and load throttled below the tipping point, the loop ran in reverse: caches filled, hit rates rose, database load fell, and the boot limit walked back up to normal while maximizing database goodput until service was fully restored.
The learnings section is where the post earns its reputation. The Consul restarts had been paused early — and it changed nothing, which is the class in one sentence. Mcrib's contribution is examined without blame and with precision: it 'contributed to the incident not by being incorrect or inefficient — quite the opposite'; it detected and repaired cache-node departures faster than the old system ever could, and that very efficiency amplified the churn — 'objectively a better system... but its efficiency made the broader system behave in a less safe way.' The scatter query was permanently moved to a table sharded by channel, and the rest of the cache-fronted query inventory was audited for the same shape of risk. The post closes with Richard Cook's 'How Complex Systems Fail' — complex systems always contain latent failures, individually harmless, waiting for the combination — and with the reason Slack publishes these at all: so the rest of the industry can learn from their worst Tuesday.
Tradeoffs
- The defining property of this failure class is that removing the cause doesn't help. Pausing the Consul rollout early was the correct move and accomplished nothing, because the system had crossed into a state that regenerates itself: empty cache drives database overload, overload times out the refill queries, and the still-empty cache drives the next round. Response plans built around 'find the trigger and stop it' are structurally insufficient here — the trigger is history; the loop is the enemy.
- A cache that makes your traffic servable is capacity, not optimization — and its warmth is a hard dependency. Slack's database tier could not serve boot traffic at realistic hit-rate losses; the fleet's true capacity included the cache being full. Systems in this position need cache warmth treated like any other provisioned resource: protected during maintenance, warmed before serving, and monitored as the load-bearing component it is.
- Better automation can make the system worse, without being wrong. Mcrib replaced departed cache nodes faster and more correctly than its predecessor — and that speed converted a slow rolling restart into rapid cache churn. Remediation speed is not free: every automatic 'repair' that costs the system something (an empty node, a flush, a failover) needs pacing, budgets, or hysteresis, or its efficiency becomes an amplifier under exactly the conditions it was built for.
- Data layouts carry latent amplifiers that only fire cold. The scatter query was invisible for as long as the cache absorbed it — immutable data, long TTLs, near-perfect hit rates — and it converted each cache miss into a query against every shard the moment the cache went cold. The audit that matters is the one Slack ran afterward: find every high-volume query whose cost is only acceptable because a cache is currently hiding it.
- Escaping a self-sustaining failure means shedding load below the tipping point — and shedding is choosing. Throttling client boots was a priority decision stated plainly: not-yet-connected users were sacrificed so connected users kept working and the cache could refill. And the return trip has its own discipline: the too-large limit increase caused a relapse; small increments with goodput watched at each step got them home. Near a tipping point, both directions are cliffs.
- Retries contributed even though they were done right. Slack's clients back off exponentially with jitter — the textbook discipline — and the post still lists automated retries among the loads that fed the failure, because a client can't distinguish a transient local failure (retry promptly) from global overload (ideally, go silent). Correct per-client behavior summed to harmful fleet behavior; the honest conclusion is that client-side discipline reduces but cannot eliminate the retry contribution to a cascade.
Patterns in this article
- Load-Bearing Cache
Minted from the incident's structural lesson: Slack's database tier could not serve boot traffic without the cache absorbing most of it — the cache was provisioned capacity wearing an optimization's clothes, and its warmth was a hard dependency that maintenance churned away. The pattern names the discipline for systems in that position: treat hit rate as capacity, protect warmth during maintenance and node replacement, warm before promoting, and audit for queries whose cost is only survivable while the cache hides it.
- Conservative Auto-Remediation
Recurs as its own anti-instance, taught by the cost of absence: Mcrib repaired cache-node departures faster and more correctly than the system it replaced, and the post's verdict is exact — 'its efficiency made the broader system behave in a less safe way.' Every automatic repair that costs something (an empty node, a flush) needs pacing or budgets; Slack's follow-up changes to the Mcrib control loop add precisely that restraint.
- Priority-Aware Load Shedding
Recurs in the storm rather than in the design phase: the client-boot throttle was an explicit priority call — users without booted clients sacrificed so connected users kept working and the refill queries could land — followed by the return-trip discipline the class demands: one too-large limit increase caused a relapse; small increments with goodput watched at each step restored full service. (Conditional slug per the round-24 protocol: agent aligns to the live pattern name.)