Pattern · seen in 3 breakdowns across 3 companies

Independent Observability

observability

Definition

Run a monitoring path that shares as little as possible with the system it watches - separate infrastructure, separate dependencies, separate assumptions - so that the failure being detected cannot also disable the detection. The strongest form is deliberately primitive: a watcher outside the walls that consumes the system's public interfaces exactly as a user would, knowing nothing about the architecture, because every shared component and every shared assumption is a channel through which the observed system's failure propagates into the observer. Its job is narrow by design: to say THAT the system is failing, within minutes, when everything richer has gone dark.

This is the positive form of a more general trap: the circular dependency. A circular dependency exists when system A monitors or controls system B but itself depends on B to function. It is invisible in normal operation and becomes visible only when B fails, at which point A loses its ability to detect or respond to that failure exactly when it matters most. The same shape recurs well beyond monitoring: deploy systems that depend on the services they deploy, secret stores whose own secrets live inside themselves, DNS that depends on DNS to resolve its upstream, certificate authorities whose renewals depend on the services the certificates protect. The discipline is to map every dependency of a system with a safety or control role and verify that none of them is the system being protected - a mapping that is usually surprising, because modern infrastructure shares so much substrate (control planes, identity, DNS, metric pipelines) that hidden loops are nearly always present.

Severing such a dependency means moving the watcher into a different failure domain: separate infrastructure, separate region, separate provider, or something primitive enough to have no shared substrate at all. The pattern accepts a two-tier structure: rich, deeply integrated in-platform telemetry for diagnosis on ordinary days, plus the poor-but-independent layer whose value shows up only on the extraordinary one. Boundary against Dead Man's Switch: that pattern derives signal from an expected heartbeat's absence; this one is an active external prober whose independence, not its mechanism, is the load-bearing property.

When it applies

01Platforms whose own monitoring, alerting, or paging runs on the platform itself - most acutely, monitoring products monitoring themselves
02Any system whose total failure must still be detected within minutes, when in-band telemetry is part of the casualty list
03Repair and control automation whose health checks run on the substrate being repaired
04Identifying root causes after outages where 'the alerts didn't fire' or 'we couldn't deploy a fix' was part of the incident - these almost always have a circular dependency underneath
05Building secrets management, certificate authorities, or identity systems where the system protecting credentials must not itself depend on those credentials
06Adding any dependency to a Tier-0 system - explicitly mapping whether the new dependency could, transitively, depend on the system being protected

Tradeoffs

Independence is bought with poverty: the external watcher can detect, not diagnose - depth still requires the in-band telemetry that shares fate
The independent path is easy to neglect precisely because it is simple and rarely fires; it needs its own testing, paging, and ownership
True independence is expensive to maintain as the platform evolves - every convenience integration quietly re-couples observer to observed
Mapping every dependency to find hidden loops is laborious, and requires either tooling or deliberate manual audit of deep chains
Some shared substrate is unavoidable (DNS, internet routing, cloud control planes); the goal is severing the dependencies that matter, not pretending zero sharing is possible
Over-severance produces so much duplication that the architecture becomes harder to maintain than the failures it prevents; the right amount depends on the stakes of the system being protected

The same move, 3 ways

Every row is a production system that bet on this pattern — the note says how, in that system's own terms.

Airbnb
Airbnb Engineering
2026
The common thread of the redesign is that no part of a safety mechanism may depend on the thing it protects. Airbnb breaks the loop at three points: dedicated clusters break the compute dependency, a separate network path breaks the data-flow dependency, and an external heartbeat watched from outside Airbnb's own infrastructure breaks the alerting dependency. Each was a hidden loop, invisible in normal operation and dangerous only when it mattered. The lesson is the discipline of mapping every such loop before deliberately cutting it. Read the breakdown →
Datadog
Datadog Engineering Blog
2023
Datadog shows the payoff of an independent observer. A deliberately basic outside watcher, sitting outside the infrastructure and assuming nothing about it, detected a total platform outage in three minutes, while the in-platform alerting died along with the platform. The rule it demonstrates: an independent observer is worth exactly as much as it does not share with the thing it watches, including its assumptions. Read the breakdown →
Roblox
Roblox Blog
2022
The monitoring that would have shown what was wrong ran on Consul, so when Consul went unhealthy the instruments went dark, and the team burned four wrong theories over fifty hours before the flame graphs finally surfaced the real bug. This is a circular dependency: the tool that watches a system must not itself depend on that system, or it fails at the exact moment it is needed. Roblox's first post-outage fix states the rule directly, that the monitoring no longer depends on the systems it watches, and the 73 hours are what its absence cost. Here the lesson was learned the expensive way, from three days in the dark. Read the breakdown →

Often used together

Patterns sharing breakdowns with this one — derived from co-occurrence, threshold ≥2 shared.

fault-isolation ×3

Problems this pattern answers

The walls where its breakdowns live — each opens the cross-company comparison.

Observer shares fate with observedAIRBNB · DATADOG · ROBLOX →