Dark Dashboards: The 73 Hours When Roblox Couldn't See Itself

Roblox runs its core infrastructure in its own data centers — over 18,000 servers and 170,000 containers orchestrated by the HashiStack: Nomad schedules work, Vault holds secrets, and Consul provides service discovery, health checks, session locking, and a KV store. Consul's role made it load-bearing for everything: when a Roblox service wants to talk to another, Consul tells it where; Nomad and Vault themselves depend on it. The post's own summary is unsparing — the system failed because Consul was a single point of failure, and Consul was not healthy.

On the afternoon of October 28, 2021, Vault performance degraded and one Consul server showed high CPU. KV write latency that normally sits under 300ms at the 50th percentile rose to 2 seconds. By 16:35 player counts had halved; then the platform went fully dark, fifty million daily players locked out behind a maintenance page.

What followed was four failed diagnoses in sequence, each consuming hours: degraded hardware (nodes replaced — no change); a traffic tipping point (the entire cluster migrated to 128-core NVMe machines — no change, and as the team later learned, the dual-socket NUMA architecture made the real problem worse); corrupted state (a full cluster shutdown and snapshot reset behind iptables — healthy briefly, then the familiar collapse); and sheer usage (services scaled from hundreds of instances to single digits, health checks stretched from 60 seconds to 10 minutes — relief for hours, then unhealthy again with a fraction of the load). Fourteen hours in, the team still didn't know what it was fighting; over fifty hours in, it was still working theories.

The reason diagnosis ran blind is the crux: critical monitoring systems that would have provided better visibility into the cause relied on affected systems — on Consul itself. The telemetry that should have illuminated the failure was a casualty of it. Only when the team stopped reasoning from the outside (hardware, traffic, load) and went spelunking in debug logs, perf reports, and flame graphs did the evidence surface: kernel spin locks concentrated in the streaming subscription code path.

The breakthrough came from Consul internals, not infrastructure. Months earlier Roblox had upgraded to Consul 1.10 for a streaming feature that cuts the CPU and bandwidth of distributing updates across large clusters; it worked well enough that it was incrementally enabled more broadly, and one day before the outage it reached the traffic-routing service — whose node count had simultaneously been increased 50% for end-of-year traffic. HashiCorp's explanation of the pathology: streaming uses fewer concurrency-control elements than long polling, and under simultaneously high read and write load, contention concentrates on a single Go channel, blocking writes — worse still on the new dual-socket NUMA machines. Disabling streaming everywhere dropped KV write p50 from 2 seconds back to 300ms.

Stability required one more workaround: some elected leaders still exhibited the old latency, for reasons unknown mid-incident, so the team pragmatically prevented the problematic servers from staying elected and moved on. The full explanation arrived days later: Consul stores its Raft log in BoltDB, whose design never shrinks the file — deleted pages go on a freelist that is written to disk with every append. Roblox's workload had grown that freelist to nearly a million page IDs: a 4.2GB log file holding 489MB of data, paying a 7.8MB freelist write for every 16kB append, backing up into full TCP buffers and zero windows on unhealthy leaders. The fix was a compaction process using existing BoltDB tooling, and, longer-term, replacing BoltDB with its successor bbolt, which bounds freelist growth.

Recovery from a fully-down state was its own engineering problem. The caching tier — 1 billion requests per second across its layers in normal operation — held only transient data, so redeployment was the path; but the deployment tooling was built for incremental adjustment of live systems, not bootstrapping from scratch, and the earlier snapshot reset had left stale scheduling data in Consul's KV, while one unhealthy node that the scheduler saw as wide open soaked up aggressive job placements that all failed. Sixty-one hours in, Consul and the caches were healthy. The final act was deliberately slow: with cold caches, the team used DNS steering to admit randomly selected players in roughly 10% increments, checking database load, cache performance, and stability at each step — dedicated players reverse-engineered the scheme on Twitter to get in early — reaching 100% at hour 73.

The remediations map one-to-one onto the anatomy of the failure. First, the circular dependency: telemetry systems no longer depend on the systems they are configured to monitor, with new targeted alerts on the specific signatures of this outage. Second, the shared-fate hub: critical services split onto dedicated Consul clusters, KV data migrated to more appropriate stores, obsolete data deleted, and a build-out of a second, geographically distinct data center with multi-AZ efforts accelerated on the roadmap. Third, the recovery gap: cache deployment redesigned for cold starts, Nomad enhancements for turning up large jobs after long unavailability. Streaming itself returns only after HashiCorp's redesign is tested at Roblox's scale — the feature wasn't wrong, its concurrency model was, and the post is careful to keep those separate.