Problem · seen at 3 companies · 2020–2023
Gray failure defeats automatic detection
It's failing or healthy depending on who you ask — so nothing automatic ever decides to act.
The wall
When a failure is partial and inconsistent, different components hold contradictory views of what is healthy, so automated detection cannot converge on a fault the system can't agree exists — and mitigation must fall to a human acting on ambiguous evidence. 3 teams hit this wall; the breakdowns below are the evidence.
Same wall, 3 systems
Slack
Slack Engineering
Gray failure in one zone defeated automatic detection: the components disagreed on what was actually down. Read the breakdown →
Meta
Engineering at Meta
A faulty chip computes wrong answers and leaves no trace in any log. Read the breakdown →
Cloudflare
The Cloudflare Blog
A switch failed partially, not fully, and every failover built for clean up-or-down deaths misfired. Read the breakdown →