Monitor the Monitors: How Datadog Watched Its Own Global Outage
On March 8, 2023 at 06:00 UTC, a routine security update rolled out automatically across Datadog's servers and, as a side effect, wiped out the network settings on each machine as it restarted. Within an hour, tens of thousands of machines across five regions and three cloud providers dropped off the network at once. For a monitoring company the outage had a cruel twist: not only did customers' monitors go silent, but Datadog's own monitoring, which runs on the same platform, went down with it. Detection came in three minutes from a simple watcher that runs completely outside Datadog's own systems, assumes nothing about how the platform is built, and monitors it from the outside the way any customer would. What follows is the story of the largest incident response in the company's history: several hundred engineers, judgment over rigid checklists, working in shifts through a recovery of about 13 hours. Running through it is one lesson: regions that share no infrastructure can still fail together.
Run the 06:00 update wave with your alerting on the platform it watches, and see that a dying monitoring system does not turn red, it goes quiet. Then switch on the outside watcher, get paged at 06:08, and replay the 13-hour recovery from computing capacity all the way to 16:44.
Problem
This problem had been waiting to happen for two years. Back in December 2020, an update to a low-level Linux networking component changed how it behaved on startup: it would now wipe out any network routing rules it did not recognize. A later version made this the default. Ubuntu 22.04 shipped with that version, and by March 2023 more than 90% of Datadog's servers were running 22.04. This mattered because the behavior was invisible in normal operation: on a fresh boot, the networking component starts before Datadog's own networking (Cilium) adds its routes, so there is nothing unfamiliar to wipe. Only restarting the component on an already-running server triggers it, and no update had forced such a restart since the migration to 22.04.
Then, on March 7, a routine security fix landed in Ubuntu's package repositories. Datadog had never turned off Ubuntu's automatic-update feature, which had run for years without trouble, even though Datadog does not rely on it (it patches by regularly replacing servers instead). By default, that feature installs updates once a day, in the 06:00 UTC hour. Between 06:00 and 07:00 on March 8, every 22.04 server that had downloaded the new update restarted the networking component, which wiped Datadog's routes, and the server fell off the network. This hit machines on all three cloud providers at once, including the very machines running the systems that would otherwise have repaired the damage.
Datadog's regions were built precisely so this could not happen everywhere at once. Each region is its own separate software stack, on independent cloud providers, with no direct connections between them and no single control system spanning them. All of that held. And all of it was bypassed by one thing nobody had modeled: an update channel that reached every region at the same time. Datadog's own words: we failed to imagine how the regions could remain indirectly related. For a monitoring company the damage went one level deeper: users could not reach the platform, their monitors were down and not alerting, and so the watcher was down, both for its customers and for itself.
Solution
Detection is something you have to set up before an outage, not after; once the platform is down, it is too late to add it. Datadog had set it up in advance. Datadog's main monitoring instruments everything and follows a 'you build it, you own it' rule, but that monitoring was mostly gone with the platform. Alongside it runs a second, deliberately basic watcher that sits completely outside Datadog's own systems, assumes nothing about how the platform is built, and checks it from the outside the way a customer would. In Datadog's words, this is how they monitor the monitors, and stay alerted even in the rare case where the platform itself is largely down, as it was on March 8. At 06:03 UTC, three minutes in, that watcher flagged a problem. At 06:08, two teams were paged: one whose automation noticed its own services were not restarting, and the team that gets paged by the outside watcher when Datadog's alerting is in trouble. The first public status-page update went out at 06:31.
The response that followed was the largest in the company's history: several hundred engineers working in shifts across many parallel video calls and workstreams. The thinking behind the response was spelled out plainly. Because a system that changes constantly makes fixed recovery checklists impossible to keep current, engineers were trusted to use their own judgment about the services they knew best, rather than follow rigid scripts. Structure came from standard incident-command practices layered on top of that judgment: clear escalation paths, discipline about severity, and shift handoffs for what they called the long haul.
The damage itself set the order of recovery: restore the raw computing capacity first, because everything else depends on it. The systems that manage the servers recovered on their own as automated replacement swapped out dead machines, so that by 08:00 all but one cluster were healthy, and the cloud recreated tens of thousands of machines within about two hours. The giant clusters (over 4,000 machines each) took longer: their automatic scaling kept piling up machines that never came fully online, hitting a 6,000-machine ceiling until engineers stepped in by hand. Key milestones followed:
- 08:30, a working fix was found for the EU1 region.
- 09:13, web access came back.
- 11:36, the automatic-update feature was identified as the trigger.
- 16:44, the first major service was declared working again.
Restoring the majority of computing capacity across all regions took roughly 13 hours, and application-level recovery continued from there.
The permanent fixes address the root causes:
- The automatic-update channel was turned off everywhere, at no cost to security, because the same updates already arrive through the safer, staged process of replacing servers.
- The networking component was reconfigured to leave the routing table alone when it restarts.
- The whole fleet was audited for other hidden channels that could change everything at once.
- Days later, the Cilium team strengthened its rules so the low-level component can no longer remove them.
The outage reshaped priorities in another way too. Customers said clearly that live data and alerts matter far more than access to historical data, a ranking of what matters most that the platform now knows it has to build in.
Tradeoffs
- The outside watcher is deliberately simple, and that simplicity is exactly what keeps it independent. Because it runs outside the infrastructure, assumes nothing about the design, and checks the platform like a user would, it shares no fate with the platform. For the same reasons, it can only say that something is wrong, never what. It bought Datadog a three-minute warning, but the actual diagnosis still had to come from engineers reasoning about a system whose usual detailed monitoring had been knocked out along with everything else.
- Trusting judgment over checklists is a bet on people, and the bet gets bigger the more novel the incident. The reasoning is that fixed procedures cannot stay current with a platform that changes constantly, so responders are trusted to improvise from expertise. That is exactly what a first-ever global outage demands. It is also exactly what makes response quality depend on fragile things: who happens to be awake, and how well each shift passes on what it knows to the next. It depends, too, on whether several hundred people can coordinate by judgment rather than by script.
- The region isolation was real, but it was aimed at the wrong layer. Independent clouds, no shared control system, separate failure boundaries: all of it worked, and all of it was bypassed by one synchronized update channel plus the fact that 90% of the servers ran the same operating system. An isolation review that checks which systems depend on which, but not which channels can change everything at once or how uniform the fleet is, signs off on a boundary the next incident will simply step around.
- Automatic security patching trades rollout discipline for speed, and Datadog's fix does not give up on fast patching; it just moves it onto a safer, staged path. The automatic-update feature applied a patch to the entire fleet within one hour, with no staged rollout, no waiting to see if problems appeared, and no limit on how much it could break at once. Turning it off cost nothing, because the same updates already arrive through the staged process of replacing servers. The general lesson: a fast update channel that duplicates a safer one is pure risk, and a fast channel you truly cannot replace is a deliberate decision about how much simultaneous change you can survive.
- Self-healing machinery repairs the ground it stands on, until it cannot. Automated replacement quietly restored the server-management systems by 08:00, automation at its best. But in the biggest clusters the same automatic scaling did exactly what it was designed to do about waiting workloads. It stacked up thousands of machines that never came fully online, until it hit the capacity ceiling and had to be overridden by hand. The very same automatic behavior was both the rescue and the obstacle. The only difference was whether the layer beneath it was healthy.
- Recovery has an order, and the order is a dependency chart, not a wish list. Computing capacity first, because everything rides on it; platform features next; application recovery last. Roughly 13 hours passed before that first leg was done, with shifts arranged for that length. Customers taught a matching lesson for the application layer: live data and alerts come before access to historical data, a ranking of importance the platform now knows it must build in.
Patterns in this article
- Independent Observability
Datadog shows the payoff of an independent observer. A deliberately basic outside watcher, sitting outside the infrastructure and assuming nothing about it, detected a total platform outage in three minutes, while the in-platform alerting died along with the platform. The rule it demonstrates: an independent observer is worth exactly as much as it does not share with the thing it watches, including its assumptions.
- Fault Isolation
The isolation here was genuine: autonomous regions, three independent clouds, no global control system. It held at the layer it was built for, but a synchronized update channel and a fleet that was 90% one operating system connected everything at a layer above it. Datadog's own line, 'we failed to imagine how the regions could remain indirectly related,' is the question every isolation design should ask of itself.
- Universal Staged Rollout
This one comes from the fix, not the failure. After the incident, Datadog turned off the old automatic-update channel, at no cost to security, because the same updates already arrive through a staged, controlled process, and audited the fleet for other channels that skip that discipline. The claim: every channel that can change the fleet, including operating-system security updates, should go through a staged rollout, because an unstaged one turns any hidden flaw into a single synchronized global failure.
Also solving this
Other systems in behindscale's Observer shares fate with observed class: