Problem · seen at 3 companies
Gray failure defeats automatic detection
It's failing or healthy depending on who you ask — so nothing automatic ever decides to act.
The wall
When a failure is partial and inconsistent, different components hold contradictory views of what is healthy, so automated detection cannot converge on a fault the system can't agree exists — and mitigation must fall to a human acting on ambiguous evidence. 3 teams hit this wall; the breakdowns below are the evidence.
Same wall, 3 systems
Slack
Slack Engineering
Gray failure in one availability zone defeated automatic detection — components disagreed on what was down. Read the breakdown →
Meta
Engineering at Meta
Defective silicon computes wrong answers and leaves no record in any log. Read the breakdown →
Cloudflare
The Cloudflare Blog
A switch failed partially, not fully, and every failover built for clean up-or-down deaths misfired. Read the breakdown →