Pattern · seen in 2 breakdowns across 2 companies

Conservative Auto-Remediation

Before an expensive automatic fix fires, we need evidence that it is actually worth its cost and time. A brief blip should not trigger an automatic costly cure.

resilience

The mechanism

The pattern at its core: an ambiguous signal that a system component might be down, and the choice between firing an expensive fix at once or waiting long enough to be sure the fix is actually needed.

LIVE ARTIFACTTHE PATTERN, GENERALOPEN FULL SCREEN ↗
THE IDEADetection during a partial failure is unreliable, so a signal that looks bad might just be a brief blip. An eager trigger fires the expensive fix on any bad signal, so it fires on blips that would have cleared on their own - a costly false alarm. A conservative trigger waits out an evidence window first, using elapsed time as evidence: blips clear inside the window and fire nothing, while a real outage is still fixed, just a little later. The costlier the fix, the longer that wait should be.
WHAT TO TRYRun an eager trigger and a conservative one against a brief blip and a real outage. The eager one fires the expensive fix on the blip - a false alarm; the conservative one lets the blip clear and still fixes the outage, at the cost of a little recovery time.

Fire an expensive fix twitchily or wait for evidence - test each against a brief blip and a real outage.

Definition

When an automatic recovery action is expensive, damaging, or hard to reverse, don't let it fire on weak evidence. Demand evidence that matches the cost of the action, and count elapsed time as part of that evidence: a brief, unclear blip should not be able to trigger an hours-long, drastic fix. When a system is only partly broken, the signals are unreliable, so the question you are really asking changes. It stops being just 'is this component down?' and becomes 'is my evidence strong enough to justify a fix this costly, given the damage it does if I've got it wrong?' The delay before the fix fires is tuned for each target, based on how costly that fix is - not on some single, system-wide 'react within N seconds' goal.

COST SETS THE DELAY
A comparison: a cheap reversible fix fires on thin evidence, an expensive irreversible fix waits for strong evidence.
The delay before an automatic fix fires is tuned to the cost of that fix. A cheap, reversible fix can be triggered on thin evidence; an expensive, hard-to-reverse one should wait for strong evidence, since a wrong firing is costly.

This pattern is about the response, not the detection. It assumes signals will sometimes be unclear or plain wrong, and limits how eagerly any signal is allowed to act. It helps to place it against two neighbors:

  • Dead Man's Switch adds a way to detect trouble - a missing heartbeat is the signal - while this pattern slows the response, no matter what detected the problem
  • Throttled Readmission paces how fast traffic returns to a service after it recovers, while this pattern paces the firing of the recovery action itself

When it applies

01The fix itself is expensive to run or to reverse. Automated failovers or promotions - rebuilding a replica, clearing a cache, changing the topology - that cost a lot whether they help or not.
02The system fails in unclear, partial ways. When the failure is only partial and the signals contradict each other, 'no signal' doesn't prove the component is actually dead.
03Firing wrongly costs more than waiting a bit. Whenever a false trigger would hurt more than pausing through a short, unclear moment, the delay before firing is cheap insurance.

Tradeoffs

A truly dead component recovers more slowly now. It has to wait out the full evidence window before the fix fires, so the safety margin is paid for in real recovery time.
One global setting becomes many local ones. Tuning the delay per target means every target needs its own value, with an owner and a reason behind it.
Slowing the trigger treats the symptom, not the cause. It doesn't make your detection any better, and it can hide signals that were actually worth looking into.

The same move, 2 ways

Every row is a production system that bet on this pattern — the note says how, in that system's own terms.

Slack
Slack Engineering
2022
Recurs as its own anti-instance, taught by the cost of absence: Mcrib repaired cache-node departures faster and more correctly than the system it replaced, and the post's verdict is exact — 'its efficiency made the broader system behave in a less safe way.' Every automatic repair that costs something (an empty node, a flush) needs pacing or budgets; Slack's follow-up changes to the Mcrib control loop add precisely that restraint. Read the breakdown →
Cloudflare
The Cloudflare Blog
2020
When a fix is expensive or hard to undo, the trigger that fires it should demand evidence proportional to the cost, and time is a form of evidence: six minutes of ambiguity should not launch an hours-long, irreversible cure. Cloudflare's response wasn't better detection (a partial failure defeats detection by definition) but a slower trigger, tuned per-database to how costly the cure is. The faster and cheaper the remedy, the twitchier the trigger can safely be; the slower and more destructive, the more it should wait. Read the breakdown →

Problems this pattern answers

The walls where its breakdowns live — each opens the cross-company comparison.

Degraded state outlives its triggerSLACK →Gray failure defeats automatic detectionCLOUDFLARE →