Pattern · seen in 2 breakdowns across 2 companies

Load-Bearing Cache

When a database can handle only the reads its cache misses, losing the cache can take the system down, not just slow it: every read it was answering now hits the database at once.

resilience

The mechanism

At its core: the database can serve only part of peak traffic on its own, so the cache's hit rate is real capacity. Let the hit rate drop too far and the database is overloaded, and the outage does not fix itself.

LIVE ARTIFACTHIT RATE IS CAPACITYOPEN FULL SCREEN ↗
THE IDEAThe database behind the cache cannot serve peak traffic on its own. It was set up to handle only the reads the cache misses, so the cache's hit rate is real capacity: while the hit rate stays high, the database sees only a trickle of reads. If the hit rate drops too far, the database is overloaded, and the failure feeds itself, because an overloaded database fails the very reads that would refill the cache. The fix is to limit how much of the cache any single maintenance job is allowed to empty, so the hit rate never drops that far.
WHAT TO TRYEmpty part of the cache with a maintenance event and watch the load on the database rise toward its limit. Find the point where it goes over and cannot recover, then turn on protection and try the same event again.

Empty a cache past the point of no return and the database goes down with it - protect the cache and it holds.

Definition

A cache starts as a speed-up: the system still works without it, just slower. At scale that can stop being true. To save money, the database behind the cache is often given only enough capacity to handle the reads the cache misses, because the cache is expected to answer most reads. The share of reads the cache answers is called its hit rate. If the hit rate drops far enough, too many reads reach the database at once and it can no longer keep up.

Now the cache is load-bearing: the system depends on it to stay up, not just to stay fast. Losing the cache is no longer a slowdown - it can take the system down. Worse, the outage often keeps itself going. Once the database is overloaded, it starts failing the very reads that would refill the cache, so the cache stays empty and the database stays overloaded.

WHY THE CACHE STAYS EMPTY
A four-step loop where an empty cache floods the database, refills fail, and the cache stays empty.
An empty cache floods the database until it is overloaded, and the overloaded database then fails the very reads that would refill the cache - so the cache stays empty and the outage keeps itself going.

The pattern is the discipline for systems where the cache has become load-bearing. It has four parts:

  • know the breaking point - treat the hit rate like a capacity limit, and find the hit rate below which the database can no longer keep up when traffic is at its peak
  • protect the cache during maintenance - routine work like replacing or restarting nodes empties part of the cache; do it in small steps, and do not clear the cache unless you have to
  • fill a node before it serves - when a node starts up empty, fill it with data before it takes real traffic, instead of putting an empty node in front of users
  • watch for hidden expensive reads - some reads are cheap only because the cache hides their cost, like reads that hit many shards or join large tables; each one gets far more expensive the moment the cache is empty

One limit on all this: if losing the cache only makes the system slower, you do not need any of it. These habits are for the case where the honest answer to 'can the database handle the traffic without the cache?' is no.

When it applies

01The database cannot serve peak traffic on its own. It was set up to handle only the reads the cache misses, so once the hit rate drops to a realistic low, more reads arrive than it can handle.
02Routine work can empty the cache fast. Replacing or restarting nodes, clearing the cache, deploys, and network blips can each wipe out a large share of it in a short time.
03Some reads are cheap only because the cache hides their cost. Reads that hit many shards or join large tables cost little while the cache answers them, but each gets far more expensive the moment the cache is empty.

Tradeoffs

Protecting the cache slows maintenance: replacing nodes in small steps, avoiding cache clears, and filling nodes before they serve all add time to work that used to be quick.
The safe hit rate is a moving target. The point where the database can no longer keep up shifts with traffic and data, so the safe hit rate has to be re-checked under peak load, not measured once and trusted.
Keeping the cache full can fight correctness. Clearing a rejoined node to get rid of stale data is the safe choice, but it also empties that node's cache. The two goals pull against each other, and the pattern forces the conflict into the open.

The same move, 2 ways

Every row is a production system that bet on this pattern — the note says how, in that system's own terms.

Reddit
r/RedditEng
2023
Second company for the round-28 mint, one round later: the readmission ramp exists because Reddit's caches are load-bearing — 'Reddit relies on a lot of caches to operate semi-efficiently,' and full traffic against cold caches produces thundering herds in downstream services that idled during the outage. (Conditional per the mint's merge-or-discard status; agent aligns.) Read the breakdown →
Slack
Slack Engineering
2022
Minted from the incident's structural lesson: Slack's database tier could not serve boot traffic without the cache absorbing most of it — the cache was provisioned capacity wearing an optimization's clothes, and its warmth was a hard dependency that maintenance churned away. The pattern names the discipline for systems in that position: treat hit rate as capacity, protect warmth during maintenance and node replacement, warm before promoting, and audit for queries whose cost is only survivable while the cache hides it. Read the breakdown →

Problems this pattern answers

The walls where its breakdowns live — each opens the cross-company comparison.

Degraded state outlives its triggerSLACK →Unrecorded config outlives its authorsREDDIT →