Problem · seen at 3 companies
Observer shares fate with observed
Your monitoring runs on the very thing it watches, so it goes dark exactly when you need it most.
The wall
A system depends on the very infrastructure it exists to watch or protect — most commonly monitoring running on the servers it monitors — so a failure in that foundation disables the safeguard exactly when it is needed. 3 teams hit this wall; the breakdowns below are the evidence.
Same wall, 3 systems
Airbnb
Airbnb Engineering
The monitoring system shared its fate with the very infrastructure it watched. Read the breakdown →
Datadog
Datadog Engineering Blog
When the platform is the observability, an outage takes your eyes with it. Read the breakdown →
Roblox
Roblox Blog
Telemetry depended on the failing Consul cluster, so responders diagnosed a 73-hour outage nearly blind. Read the breakdown →