Problem · seen at 3 companies · 2022–2026
Observer shares fate with observed
Your monitoring runs on the very thing it watches, so it goes dark exactly when you need it most.
The wall
A system depends on the very infrastructure it exists to watch or protect — most commonly monitoring running on the servers it monitors — so a failure in that foundation disables the safeguard exactly when it is needed. 3 teams hit this wall; the breakdowns below are the evidence.
Same wall, 3 systems
Airbnb
Airbnb Engineering
The monitoring system shared its fate with the very infrastructure it watched. Read the breakdown →
Datadog
Datadog Engineering Blog
When your monitoring runs on the very platform it watches, an outage blinds you too. Read the breakdown →
Roblox
Roblox Blog
The monitoring depended on the failing Consul cluster, so responders diagnosed a 73-hour outage nearly blind. Read the breakdown →