Generic Mitigation

resilience

Definition

Build mitigations that can be applied before the root cause is understood: actions that reduce impact for a broad class of failures, are safe to apply experimentally, and are cheap to reverse. A generic mitigation is judged on four properties — it acts fast relative to the availability budget, it introduces no user-visible errors of its own, it can be applied incrementally so recovery can be tested at small percentages, and it does not depend on resources inside the failure domain it mitigates.

The pattern's premise is epistemic humility during incidents: diagnosis is slow and often wrong under pressure, while a small set of reversible actions — drain the zone, roll back the deploy, fail over the region, disable the feature flag — resolves most incidents without anyone knowing the cause yet. Making those actions safe to try on suspicion converts incident response from a diagnosis race into an experiment loop: apply, observe, keep or revert.

When it applies

  • Failure modes that defeat automatic detection (gray failures, partial network faults) where human judgment must act on ambiguous evidence
  • Services with tight availability budgets where mitigation speed matters more than root-cause speed
  • Any system where the common mitigations (drain, rollback, failover, flag-off) can be made harmless enough to apply experimentally

Tradeoffs

  • Generic means blunt: the mitigation over-scopes by design, evacuating healthy workloads along with sick ones
  • Safety properties must be engineered and continuously verified — a drain that causes errors, or depends on the failing domain, silently stops being generic
  • Optimizing for recovery time over prevention is a budget allocation: the same failure will happen again, cheaper

Seen in

  • Slack EngineeringAug 22, 2023

    The Drain Button: Slack's Migration to a Cellular Architecture

    The drain button is the post's textbook instance and it cites the concept by name: a mitigation applicable while the root cause is still unknown, safe to apply experimentally (drain; observe; undrain if unhelpful) because it adds no errors of its own. The four design goals — fast, harmless, incremental, independent of the failing domain — are effectively the pattern's requirements written down.