Pattern · seen in 3 breakdowns across 3 companies

Generic Mitigation

A generic mitigation is a safe, reversible action that reduces the damage for many kinds of failure, so you can apply it before you even know what caused the incident.

resilience

The mechanism

The pattern at its core: an incident whose cause is still unknown, and a choice between debugging for the cause first or applying a safe, reversible mitigation right away.

LIVE ARTIFACTTHE PATTERN, GENERALOPEN FULL SCREEN ↗
THE IDEAFinding the root cause of an incident is slow and often wrong under pressure, and users see errors the whole time you are looking. A generic mitigation - drain a bad zone (stop its traffic), roll back, fail over, flag off - stops the errors for a broad class of failures without needing the cause. Because it is reversible, a wrong guess just gets undone. Acting early cuts the total user impact sharply.
WHAT TO TRYRun the incident by diagnosing first, then by mitigating first. Diagnosing keeps errors high until the cause is found; a generic mitigation stops them almost immediately, at a fraction of the total user impact, before the cause is even known.

Meet an incident of unknown cause - diagnose first, or mitigate on suspicion, and watch the user impact shrink.

Definition

A generic mitigation is a fix you can apply before you know the root cause: something that reduces the damage across a whole range of failures, is safe to try, and is cheap to undo. A good one has four properties:

  • fast - it takes effect quickly, before the outage does too much damage
  • harmless - it adds no errors of its own, so trying it can't make things worse
  • gradual - you can turn it on a little at a time and watch whether things recover
  • independent - it doesn't rely on anything inside the part that is failing

The idea rests on being honest about what you don't know during an incident: figuring out the root cause is slow and, under pressure, often wrong. Meanwhile a handful of reversible moves - drain the zone (stop sending it new traffic), roll back the deploy, fail over to another region, turn off the feature flag - clears up most incidents before anyone knows the cause. Once those moves are safe to try on a hunch, incident response stops being a race to diagnose and becomes a simple loop: apply a mitigation, watch what happens, then keep it or undo it.

THE EXPERIMENT LOOP
A loop: apply a mitigation, observe the impact, then keep it if it helped or revert and try another.
Because each mitigation is safe and reversible, you don't need the cause first. Apply one on suspicion, watch whether the impact drops, then keep it if it helped or revert and try another.

When it applies

01The failure hides from your automatic detection. Gray failures and partial network faults leave the signals ambiguous, so a human has to act on a hunch rather than a clear alert.
02You can't afford to wait for a diagnosis. A tight availability target leaves very little room: 99.99% allows only about 52 minutes of downtime across the whole year. Every minute spent diagnosing instead of mitigating eats that budget, so stopping the bleeding fast matters more than finding the exact cause.
03The usual moves can be made safe to try. If drain, rollback, failover, and flag-off can be made harmless enough, you can apply them on suspicion instead of waiting for certainty.

Tradeoffs

Generic means blunt: it deliberately casts a wide net, so healthy work gets moved out along with the failing work.
The safety has to be built and constantly re-checked. A drain that throws errors, or that secretly depends on the failing part, quietly stops being safe to apply on a hunch.
Fast recovery instead of prevention is a trade, not a cure: the same failure will happen again, just cheaper each time.

The same move, 3 ways

Every row is a production system that bet on this pattern — the note says how, in that system's own terms.

Slack
Slack Engineering
2023
The drain button is the post's textbook instance and it cites the concept by name: a mitigation applicable while the root cause is still unknown, safe to apply experimentally (drain; observe; undrain if unhelpful) because it adds no errors of its own. The four design goals — fast, harmless, incremental, independent of the failing domain — are effectively the pattern's requirements written down. Read the breakdown →
Cloudflare
The Cloudflare Blog
2020
The two levers Cloudflare pulled, shedding discretionary load and steering work to untouched capacity, both share a useful property: they don't require understanding the failure to help. You shed deferrable work to buy the primary headroom, and you move reads to a data center the incident never reached. Slack had pre-built its version of this as a ready button it could press on day one; Cloudflare improvised both levers mid-incident and then made automating the read-steering part of its roadmap, which is the same move as pre-positioning the button. Read the breakdown →
Shopify
Shopify Engineering
2018
Third company, and the pre-positioned form: Pod Mover is one big lever — evacuate the pod to its recovery data center, in a minute, without dropping requests or jobs — that works regardless of what is wrong with the pod's home. The recurrence arc across the class: Cloudflare improvised its levers mid-incident, Slack pre-built the drain button, Shopify rehearses its lever daily, which is what turns a mitigation into a routine. Read the breakdown →

Often used together

Patterns sharing breakdowns with this one — derived from co-occurrence, threshold ≥2 shared.

fault-isolation ×3cell-architecture ×2

Problems this pattern answers

The walls where its breakdowns live — each opens the cross-company comparison.

Gray failure defeats automatic detectionCLOUDFLARE · SLACK →Blast radius scales with cluster sizeSHOPIFY →