Problem · seen at 1 company · 2022
Degraded state outlives its trigger
Whatever triggered the outage is long gone, but the outage now feeds itself and won't let go.
The wall
A failure that persists after its trigger is removed, because the degraded state regenerates itself; exits are external (shed load below the tipping point, or add capacity).
Same wall, 1 system
Slack
Slack Engineering
Slack paused the trigger early, but by then the failure was sustaining itself. Read the breakdown →