Dark Dashboards: The 73 Hours When Roblox Couldn't See Itself

Roblox Blog Jan 20, 2022Read original

Independent Observability Fault Isolation Throttled Readmission

Starting October 28, 2021, Roblox (fifty million daily players on 18,000 self-managed servers and 170,000 containers) went fully down for 73 hours. What turned deep bugs into three days was structural. Everything ran through one Consul cluster, the coordination layer the whole platform leaned on to find services, check their health, hold locks, and store configuration. The monitoring that would have exposed the problem ran on that same cluster, so responders worked four wrong theories nearly blind. The bugs themselves were two mostly unrelated faults buried deep inside Consul: a new streaming feature that jammed under heavy reads and writes at the same time, and a storage defect that quietly turned 16kB disk writes into 7.8MB ones. Recovery became its own engineering project. It took a state reset, turning streaming off, blocking slow leaders from being elected, rebuilding from cold a cache tier that normally serves a billion requests a second, and letting players back in 10% at a time by steering DNS.

The crux

Two deep, mostly unrelated faults hid inside the one cluster that all of Roblox depended on for service discovery (how one service finds the address of another), health checks, scheduling, and secrets. The first: Consul's new streaming feature funneled too much work through a single Go channel (the queue-like pipe that Go's lightweight threads use to hand work to each other) under high read-and-write load, so writes backed up waiting for it. The second: for every small update to its running change-log, Consul was writing an extra 7.8MB of bookkeeping to disk. But the reason this became a 73-hour outage rather than a hard afternoon was circular. The monitoring that would have shown what was wrong ran on Consul itself, so the team triaged a system-wide collapse with its instruments dark. Roblox is explicit that the diagnosis difficulty was largely responsible for the length, and that the dependence of the monitoring on the failing system severely hampered triage. The observer had gone down with the observed, exactly when it was needed most.

Observer shares fate with observed

Interactive

Triage the outage with your dashboards dark, burning hours on the same four wrong theories Roblox did, then flip on independent monitoring and watch the flame graphs point straight at the answer. Finish by readmitting fifty million players without relapsing.

Open the visualization ↓

Problem

Roblox runs its core infrastructure in its own data centers: over 18,000 servers and 170,000 containers, managed by a set of HashiCorp tools known together as the HashiStack. Three of those tools matter here. Nomad schedules the work, Vault holds the secrets, and Consul provides service discovery (letting one service find another's address), health checks, locking, and a key-value store. Consul's role put it underneath everything: when a Roblox service wants to talk to another, Consul tells it where to find it, and Nomad and Vault themselves depend on Consul too. Roblox's own summary is blunt: the system failed because Consul was a single point of failure, and Consul was not healthy.

Services, Nomad, Vault, and monitoring all depend on one Consul cluster, so its failure leaves the team blind. — One Consul cluster sat under everything: services found each other through it, Nomad and Vault depended on it, and the monitoring ran on it too. So when Consul went unhealthy, the dashboards went dark and the team triaged nearly blind.

On the afternoon of October 28, 2021, Vault slowed down and one Consul server showed high CPU. Write latency on Consul's key-value store, normally under 300ms for the median request, climbed to 2 seconds. By 16:35 player counts had halved. Then the platform went fully dark, fifty million daily players locked out behind a maintenance page.

What followed was four failed diagnoses in a row, each costing hours:

Degraded hardware: nodes replaced, no change.
A traffic tipping point: the whole cluster moved to bigger 128-core machines, no change, and as the team learned later, those machines' two-processor layout actually made the real problem worse.
Corrupted state: a full shutdown and reset from a snapshot, healthy briefly, then the same collapse.
Sheer load: services scaled from hundreds of instances down to single digits, health checks stretched from 60 seconds to 10 minutes, relief for a few hours, then unhealthy again under a fraction of the load.

Fourteen hours in, the team still did not know what it was fighting. Over fifty hours in, it was still working theories.

The reason diagnosis ran blind is the crux: the monitoring that would have shown the cause ran on the very systems that were failing, on Consul itself. The monitoring that should have revealed the failure had failed along with it. The team stopped reasoning from the outside (hardware, traffic, load) and went digging in debug logs, performance reports, and flame graphs (visual breakdowns of where a program spends its time). Only then did the evidence surface: the processors were spending nearly all their time stuck waiting on locks in the streaming code.

73 hours

of full platform outage - diagnosis challenges were largely responsible for the duration

Solution

The breakthrough came from inside Consul, not from the infrastructure around it. Months earlier Roblox had turned on a new Consul streaming feature that cuts the CPU and bandwidth needed to push updates across a big cluster. It worked well, so they rolled it out more widely, and one day before the outage it reached the traffic-routing service, whose machine count had just been raised 50% for the end-of-year traffic. HashiCorp's explanation of the fault: streaming uses fewer of Go's coordination pipes (channels) than the older method did. Under high read and write load at the same time, all the contention piles onto a single one of those channels, and writes back up waiting behind it. It was worse still on the new 128-core machines, which have two physical processors with a memory layout (called NUMA, where each processor reaches its own memory faster than the other's) that makes fighting over shared resources more expensive. Turning streaming off everywhere dropped median key-value write latency from 2 seconds back to 300ms.

Two faults: streaming funnels writes through one Go channel, and storage writes 7.8MB per 16kB; latency hits 2 seconds. — Two unrelated faults drove write latency from 300ms to 2 seconds. Streaming funneled work through a single Go channel, so writes backed up under heavy load; and the storage layer wrote 7.8MB of bookkeeping per 16kB of real data.

Stability needed one more workaround. Some newly elected leader nodes still showed the old slowness, for reasons nobody understood mid-incident, so the team pragmatically stopped those servers from staying elected and moved on. The full explanation arrived days later. Consul keeps a running log of changes, called the Raft log, in a storage library called BoltDB. BoltDB's file never shrinks: deleted entries leave free pages behind, tracked in a list that is written to disk on every append. Roblox's workload had grown that free-page list to nearly a million entries, so a 4.2GB log file held only 489MB of real data and paid a 7.8MB bookkeeping write for every 16kB append. That backed up into full network buffers on the unhealthy leaders, where the receiving side tells senders to stop because its buffer is full. The fix was a compaction process using existing BoltDB tooling, and, longer-term, replacing BoltDB with its successor bbolt, which keeps that free-page list from growing without bound.

Recovery from a fully-down state was its own engineering problem. The caching tier normally serves a billion requests a second across its layers. It held only temporary data, so the plan was to redeploy it from scratch, but the deployment tooling was built for adjusting live systems, not starting one from nothing. The earlier state reset had also left stale scheduling data in Consul's store, and one unhealthy node that the scheduler saw as wide open kept soaking up job placements that all failed. Sixty-one hours in, Consul and the caches were healthy. The final act was deliberately slow. With the caches still cold, the team used DNS steering, pointing some players' lookups at the real site and the rest at the maintenance page, to admit randomly chosen players in roughly 10% increments. They checked database load, cache performance, and stability at each step, and reached 100% at hour 73. Some dedicated players reverse-engineered the scheme on Twitter to get in early.

A ramp from 0 to 100% in roughly 10% steps from hour 61 to 73, admitting players via DNS steering. — With caches still cold at 61 hours, a sudden flood could knock the system back down. So the team used DNS steering to admit players in roughly 10% steps, checking load at each one, reaching 100% at hour 73.

The fixes map one-to-one onto the anatomy of the failure:

The circular dependency: the monitoring systems no longer run on the systems they watch, with new alerts on the specific warning signs of this outage.
The shared-fate hub: critical services split onto their own dedicated Consul clusters, key-value data moved to more suitable stores, obsolete data deleted, and a second, geographically separate data center built out, with multi-region work pulled forward on the roadmap.
The recovery gap: cache deployment redesigned so it can start from cold, and scheduler improvements for bringing up large jobs after a long outage.

Streaming itself returns only after HashiCorp's redesign is tested at Roblox's scale. The feature was not wrong; the way it coordinated work was, and Roblox is careful to keep those two separate.

7.8 MB

of BoltDB freelist written to disk for every 16kB Raft append - a 4.2GB log file holding 489MB of data

~10%

player readmission increments via DNS steering against cold caches, checked at every step

Open in full

Tradeoffs

A single Consul cluster for everything is simpler to run, but that simplicity means a failure anywhere takes down everything. One cluster to run, secure, and upgrade sat underneath discovery, health, locking, scheduling, and secrets. Roblox names the cost plainly: it made bugs that would have taken down a slice in isolated clusters take down the whole platform instead. The fix, dedicated clusters per critical service, buys isolation back at the price of the very extra operational work the single cluster was there to avoid.
Monitoring that shares infrastructure with what it monitors is cheap right up until it is priceless. Running the monitoring on the same infrastructure saves you from building and running a second, separate system, and it guarantees the instruments go dark in exactly the incidents where a few minutes of visibility would be worth days. Roblox's post-outage rule is the general lesson: the thing that watches a system must not depend on that system. That is a rule you can adopt before an outage teaches it to you, or after.
Efficiency features move contention, they do not erase it. Streaming genuinely cut CPU and bandwidth by using fewer of Go's coordination channels, which under high read and write load at the same time concentrated all the contention onto a single channel. Months of gradual rollout tested the feature under every condition except the one that mattered, and HashiCorp's own benchmarks at similar scale had never combined that many streams with that much churn. Whatever load condition you never tested is the one your live rollout eventually finds for you.
Bigger hardware is a diagnosis with a failure mode of its own. Moving to 128-core machines was reasonable under the traffic theory, but their two-processor layout amplified the real problem, which was processes fighting over shared resources. When the bottleneck is coordination rather than capacity, adding capacity adds more things to coordinate. The team's later reversal to 64-core machines is the quiet admission that reaching for bigger machines is a guess to check, not an automatic fix.
The snapshot reset traded a known small loss for an unknown one later. Resetting the cluster's state cost a little system-config data, which was acceptable and could be restored by hand. But it also left stale scheduling data behind, and that data came back to bite two days later, misleading the cache redeployment during recovery. The lesson: an emergency fix can carry a second, delayed cost that only shows up after the immediate crisis is over, so the accounting is not done when the system comes back up.
Owning the whole stack made the 73 hours possible and the fix possible. Roblox is honest about the bet: running its own hardware buys cost control, latency, and consistency at the price of owning every layer of a failure like this one, including running its own Consul instead of renting a managed one. That same ownership is what let Roblox profile Consul's internals side by side with HashiCorp and ship a BoltDB fix. Owning the stack cuts both ways, and Roblox does not pretend otherwise.

Patterns in this article

Independent Observability
The monitoring that would have shown what was wrong ran on Consul, so when Consul went unhealthy the instruments went dark, and the team burned four wrong theories over fifty hours before the flame graphs finally surfaced the real bug. This is a circular dependency: the tool that watches a system must not itself depend on that system, or it fails at the exact moment it is needed. Roblox's first post-outage fix states the rule directly, that the monitoring no longer depends on the systems it watches, and the 73 hours are what its absence cost. Here the lesson was learned the expensive way, from three days in the dark.
Fault Isolation
One Consul cluster carried every workload, so two deep bugs in it took down the whole platform instead of a slice. The fixes are fault isolation applied at three levels: critical services split onto their own dedicated Consul clusters, key-value workloads moved off the shared store, and a second, geographically separate data center built out, with multi-region work pulled forward. The common thread is that a shared hub trades operational simplicity for a blast radius equal to everything on it.
Throttled Readmission
The final twelve hours are this pattern in pure form: with the caches cold and confidence only partial, DNS steering let players back in about 10% at a time, checking database load, cache performance, and stability at each step. The slow ramp was the whole point, because a sudden flood of full traffic could have pushed the barely-healed system straight back into failure.

Also solving this

Other systems in behindscale's Observer shares fate with observed class: