Dark Dashboards: The 73 Hours When Roblox Couldn't See Itself
Starting October 28, 2021, Roblox (fifty million daily players on 18,000 self-managed servers and 170,000 containers) went fully down for 73 hours. What turned deep bugs into three days was structural. Everything ran through one Consul cluster, the coordination layer the whole platform leaned on to find services, check their health, hold locks, and store configuration; and the monitoring that would have exposed the problem ran on that same cluster, so responders worked four wrong theories nearly blind. The bugs themselves were two mostly unrelated faults buried deep inside Consul: a new streaming feature that jammed under simultaneously heavy reads and writes, and a storage-layer defect that quietly turned 16kB disk writes into 7.8MB ones. Recovery became its own engineering project: a state reset, disabling streaming, blocking slow leaders from being elected, rebuilding a cache tier that normally serves a billion requests a second from a cold start, and letting players back in 10% at a time by steering DNS.
Triage the outage with your dashboards dark — burn hours on the same four wrong theories Roblox did — then flip on independent telemetry and watch the flame graphs point straight at the answer. Finish by readmitting fifty million players without relapsing.
Problem
Roblox runs its core infrastructure in its own data centers: over 18,000 servers and 170,000 containers, managed by a set of HashiCorp tools called the HashiStack. Nomad schedules the work, Vault holds the secrets, and Consul provides service discovery (letting one service find another's address), health checks, locking, and a key-value store. Consul's role made it load-bearing for everything: when a Roblox service wants to talk to another, Consul tells it where to find it, and Nomad and Vault themselves depend on Consul too. The post's own summary is blunt: the system failed because Consul was a single point of failure, and Consul was not healthy.
On the afternoon of October 28, 2021, Vault slowed down and one Consul server showed high CPU. Write latency on Consul's key-value store, normally under 300ms for the median request, climbed to 2 seconds. By 16:35 player counts had halved; then the platform went fully dark, fifty million daily players locked out behind a maintenance page.
What followed was four failed diagnoses in a row, each costing hours. Degraded hardware (nodes replaced, no change). A traffic tipping point (the whole cluster migrated to bigger 128-core machines, no change, and as the team learned later, those machines' two-processor layout actually made the real problem worse). Corrupted state (a full shutdown and reset from a snapshot, healthy briefly, then the same collapse). And sheer load (services scaled from hundreds of instances down to single digits, health checks stretched from 60 seconds to 10 minutes, relief for a few hours, then unhealthy again under a fraction of the load). Fourteen hours in, the team still didn't know what it was fighting; over fifty hours in, it was still working theories.
The reason diagnosis ran blind is the crux: the monitoring that would have shown the cause ran on the very systems that were failing, on Consul itself. The telemetry that should have shown the failure had failed along with it. Only when the team stopped reasoning from the outside (hardware, traffic, load) and went digging in debug logs, performance reports, and flame graphs did the evidence surface: the processors were spending nearly all their time stuck waiting on locks in the streaming code path.
Solution
The breakthrough came from inside Consul, not from the infrastructure around it. Months earlier Roblox had turned on a new Consul streaming feature that cuts the CPU and bandwidth needed to push updates across a big cluster. It worked well, so they rolled it out more widely, and one day before the outage it reached the traffic-routing service, whose machine count had just been increased 50% for the end-of-year traffic. HashiCorp's explanation of the fault: streaming uses fewer of Go's coordination pipes (channels) than the older method did. Under simultaneously high read and write load, all the contention piles onto a single one of those channels, and writes back up waiting behind it. It was worse still on the new 128-core machines, which have two physical processors with a memory layout (NUMA, where each processor reaches its own memory faster than the other's) that makes shared-resource contention more expensive. Turning streaming off everywhere dropped median key-value write latency from 2 seconds back to 300ms.
Stability needed one more workaround. Some newly elected leader nodes still showed the old slowness, for reasons nobody understood mid-incident, so the team pragmatically stopped those servers from staying elected and moved on. The full explanation arrived days later. Consul keeps a running log of changes (the Raft log, named for the consensus algorithm Consul uses to keep every copy of the cluster's state in agreement) in a storage library called BoltDB, whose file never shrinks: deleted entries leave free pages behind, tracked in a list that is written to disk on every append. Roblox's workload had grown that free-page list to nearly a million entries, so a 4.2GB log file held only 489MB of real data and paid a 7.8MB bookkeeping write for every 16kB append. That backed up into full network buffers on the unhealthy leaders (the receiving side telling senders to stop because its buffer was full, a TCP 'zero window'). The fix was a compaction process using existing BoltDB tooling, and, longer-term, replacing BoltDB with its successor bbolt, which keeps that free-page list from growing without bound.
Recovery from a fully-down state was its own engineering problem. The caching tier (a billion requests a second across its layers in normal operation) held only temporary data, so the plan was to redeploy it from scratch, but the deployment tooling was built for adjusting live systems, not bootstrapping from nothing. The earlier state reset had also left stale scheduling data in Consul's store, and one unhealthy node that the scheduler saw as wide open kept soaking up job placements that all failed. Sixty-one hours in, Consul and the caches were healthy. The final act was deliberately slow: with the caches still cold, the team used DNS steering (pointing some players' lookups at the real site and the rest at the maintenance page) to admit randomly chosen players in roughly 10% increments, checking database load, cache performance, and stability at each step (dedicated players reverse-engineered the scheme on Twitter to get in early), reaching 100% at hour 73.
The fixes map one-to-one onto the anatomy of the failure. First, the circular dependency: the monitoring systems no longer run on the systems they watch, with new alerts on the specific signatures of this outage. Second, the shared-fate hub: critical services split onto their own dedicated Consul clusters, key-value data moved to more appropriate stores, obsolete data deleted, and a second, geographically separate data center built out with multi-region work pulled forward on the roadmap. Third, the recovery gap: cache deployment redesigned so it can cold-start, and scheduler improvements for turning up large jobs after a long outage. Streaming itself returns only after HashiCorp's redesign is tested at Roblox's scale. The feature wasn't wrong, its concurrency model was, and the post is careful to keep those separate.
Tradeoffs
- A single Consul cluster for everything is simpler to run, but that simplicity means a failure anywhere takes down everything. One cluster to run, secure, and upgrade sat underneath discovery, health, locking, scheduling, and secrets, and the post names the cost plainly: it made bugs that would have taken down a slice in isolated clusters take down the whole platform instead. The fix (dedicated clusters per critical service) buys isolation back at the price of the very operational multiplication the single cluster was there to avoid.
- Monitoring that shares infrastructure with what it monitors is cheap right up until it's priceless. Running telemetry on the same infrastructure saves you from building and running a second, separate monitoring system, and guarantees the instruments go dark in exactly the incidents where a few minutes of visibility would be worth days. Roblox's post-outage rule is the general lesson: the observer must not depend on the observed. Airbnb reached the same conclusion from a far smaller outage, and the lesson held across a 400x difference in how long the two outages lasted.
- Efficiency features move contention, they don't erase it. Streaming genuinely cut CPU and bandwidth, by using fewer of Go's coordination channels, which under simultaneously high read and write load concentrated all the contention onto a single channel. Months of gradual rollout tested the feature under every condition except the one that mattered, and HashiCorp's own benchmarks at similar scale had never combined that many streams with that much churn. Whatever load condition you never tested is the one your live rollout eventually finds for you.
- Bigger hardware is a diagnosis with a failure mode of its own. Moving to 128-core machines was reasonable under the traffic theory, but their two-processor NUMA layout amplified the real problem, which was shared-resource contention. When the bottleneck is coordination rather than capacity, adding capacity adds coordinators. The team's later reversal to 64-core machines is the quiet admission that reaching for bigger machines is a guess to check, not an automatic fix.
- The snapshot reset traded a known small loss for an unknown one later. Resetting the cluster's state cost a little system-config data, which was acceptable and could be restored by hand. But it also left stale scheduling data behind, and that data came back to bite two days later, misleading the cache redeployment during recovery. The lesson: an emergency fix can have a second, delayed cost that only shows up after the immediate crisis is over, so the accounting isn't done when the system comes back up.
- Owning the whole stack made the 73 hours possible and the fix possible. The post is honest about the bet: running on-prem buys cost control, latency, and consistency at the price of owning every layer of a failure like this one, including running your own Consul instead of renting a managed one. That same ownership is what let Roblox flame-graph Consul's internals side by side with HashiCorp and ship a BoltDB compaction. Owning the stack cuts both ways, and the post doesn't pretend otherwise.
Patterns in this article
- Independent Observability
The monitoring that would have shown what was wrong ran on Consul, so when Consul went unhealthy the instruments went dark and the team burned four wrong theories over fifty hours before the flame graphs finally surfaced the real bug. This is a circular dependency: the tool that watches a system must not itself depend on that system, or it fails at the exact moment it is needed. Roblox's first post-outage fix states the rule directly, that telemetry no longer depends on the systems it monitors, and the 73 hours are what its absence cost. Airbnb reached the same conclusion from a far smaller outage by redesigning its monitoring to sever those dependencies; here the lesson was learned the expensive way, from three days in the dark.
- Fault Isolation
One Consul cluster carried every workload, so two deep bugs in it took down the whole platform instead of a slice. The fixes are fault isolation applied at three grains: critical services split onto their own dedicated Consul clusters, key-value workloads moved off the shared store, and a second, geographically separate data center built out with multi-region work pulled forward. The through-line is that a shared hub trades operational simplicity for a blast radius that equals everything on it.
- Throttled Readmission
The final twelve hours are this pattern in pure form: with the caches cold and confidence only partial, DNS steering let players back in about 10% at a time, checking database load, cache performance, and stability at each step. The slow ramp was the whole point, because a sudden flood of full traffic could have pushed the barely-healed system straight back into failure.
Also solving this
Other systems in behindscale's Observer shares fate with observed class: