Neither Dead nor Alive: Cloudflare's Real-World Byzantine Failure
Cloudflare's control plane is the part that configures and manages the service (its API and dashboard), not the network that carries customer traffic. On November 2, 2020, it was degraded for six hours and 33 minutes, with API success dipping to 75% and the dashboard up to 80 times slower. The trigger lasted only six minutes. One network switch entered a half-working state: healthy enough that failover never fired, but broken enough that one node of etcd lost contact with its leader while every other path stayed up. etcd is the shared store the control plane uses to keep its coordination data in agreement. Three etcd nodes then held three views of reality, tied elections made the cluster read-only, and the database manager read the silence as a dead primary and automatically promoted a standby copy. A known defect then forced a rebuild of every standby, leaving a read-heavy login database on a single, overwhelmed primary for hours. Every layer had a backup; every layer's backup assumed a component is either fully up or fully down.
Kill the switch dead and watch failover shrug it off. Then set it half-alive, watch three etcd nodes hold three realities, and decide how fast the promotion trigger fires. Mitigate by hand, or set the trigger past six minutes and meet the outage that did not happen.
Problem
Cloudflare's control plane runs as many small services, duplicated across two regions. Cloudflare lists its backup discipline deliberately:
- two active switches per rack, either able to carry the load alone,
- critical services spread across racks,
- dual power feeds,
- and, for anything critical, either RAID 10 disks or data copied across at least three machines.
Backup at each layer is reviewed and required. So, Cloudflare asks, how could things possibly go wrong?
At 14:43 UTC the switch answered. Two of its protocols stayed alive: one that bundles several physical links so they act as one and spreads traffic across them, and one that announces to the network which routes the switch can carry. A third protocol, the one that makes a pair of switches look like a single switch, went down, and the switch's forwarding hardware quietly stopped handling some fraction of the packets it received. This is the failure mode backup audits do not model. The audit asks 'what if this device dies?', and the paired switch is the answer. Nothing asks 'what if it keeps claiming to be alive while doing only part of its job?', because the trigger for failover is the device dying. Six minutes later the switch recovered on its own. The impact was just beginning.
The rack held one member of the etcd cluster that Cloudflare's core data centers use to coordinate. Because the bundled links spread each server's traffic across several paths, the damage landed on exactly one path: the link from node 1 to node 3, and node 3 was the cluster's leader at the time. Node 1, unable to reach the leader, repeatedly called elections and voted for itself; node 2, which could still see node 3, kept voting for node 3. The ties never produced a leader that node 1 could reach, and because these elections block writes until they settle, the cluster went read-only for the switch's six-minute spell. Then the chain fired. With etcd read-only, two database clusters could no longer confirm they had a healthy primary. The automation drew the crash-stop conclusion, that no signal means a dead primary, and promoted each cluster's standby copy to take over, even though the real primary was fine. That standby copy is one kept exactly in step with the primary, as opposed to the other copies that lag slightly behind.
Solution
The promotion itself worked flawlessly, with no error and no data loss. But a known defect in the cluster manager required every standby copy to be rebuilt after a promotion, and the rebuild took longer the larger the database. One cluster shrugged it off. The other was the login database behind API calls and dashboard logins: read-heavy, and leaning on one standby copy that deliberately soaked up reads to spare the primary. Promotion destroyed that arrangement, and the new primary took the entire load alone, hours after the switch was already healthy again.
The response was manual, and its shape is the lesson: every effective action was a human working around the automation.
- Shed load. The team dialed back SSL certificate pushes, emails, and other work that could wait, buying the primary headroom for the hours the rebuild would take regardless.
- Use a backup the automation did not know about. Every cluster keeps live copies in the secondary data center, and because those copies are not part of the local failover process, the misfiring automation never touched them. Sending reads to them was not automated, so the team steered by hand the API traffic that could tolerate the trip, and availability improved substantially. The backup nobody had automated was the backup that survived.
- Accept a tradeoff they could not avoid. Dashboard sessions live in a Redis store in the primary data center and cannot be served from the other region, so the very actions that helped the API made the dashboard worse.
After a stretch of increasingly frustrated users, the team sent logins back to the primary region and kept the primary database running as best they could until the rebuild finished. At 21:20 UTC the first standby copy finished rebuilding and performance returned instantly.
Cloudflare's own observation is the thesis: each system had a backup, and no system fully failed. Each one entered a degraded state instead, and that combination is much harder to plan for. The fix worth borrowing was to revisit how quick the automatic response should be. Fast promotion had been a point of pride, built because doing it by hand once took painfully long, but the incident inverted that pride. For at least one database, the cure may be worse than the disease, and the trigger was retuned immediately. Cloudflare closes by defending RAFT, the relatively simple agreement algorithm etcd uses, over full Byzantine fault tolerance (BFT), a far more complex family of algorithms built to keep working even when some nodes actively lie or send conflicting information. Most cluster management skips BFT because a simple protocol with a rare, known hole is safer than a complex one that is hard to get right. And in a note added after publication, Cloudflare accepted the community's correction. This was an omission fault (a node that just goes silent on one path), not a true Byzantine fault (a node sending actively conflicting information). And omission faults can be handled without going all the way to BFT. The title overclaimed; the postscript said so.
Tradeoffs
- Choosing RAFT over Byzantine fault tolerance is a deliberate trade of safety-against-a-rare-fault for simplicity, and Cloudflare re-signs it even after being burned. A crash-stop algorithm like RAFT is easier to build, check, and debug; the price is a known weakness to exactly the conflicting-views case that occurred. Cloudflare defends the choice even after the incident: a simple protocol with a rare, known hole beats a complex one you cannot trust yourself to implement. The postscript strengthens the case, since the actual fault (an omission, one node silently unreachable) can be handled without going all the way to BFT.
- Fast automatic recovery trades the quality of the evidence for the speed of the reaction. Automatic primary promotion on a minute's notice was built deliberately, because slow promotion by hand had hurt before. The same hair trigger, pointed at six minutes of ambiguous silence, ran a promotion nothing needed, and that promotion, through the rebuild defect, was the entire outage. Cloudflare's fix was not to remove the automation but to slow it, so the evidence has time to settle. How fast to apply a cure is a setting, and its safe value depends on how expensive the cure is.
- Backup audits assume failure is all-or-nothing, and every layer's answer assumed that same question. Paired switches answer 'what if the switch dies'; RAFT answers 'what if the leader dies'; auto-promotion answers 'what if the primary dies.' A component that half-dies falls through all three answers at once, and because the bundled links spread each server's traffic thin, the packet loss was spread thin enough that no single node ever saw a fully dead link. Cloudflare's backup inventory reads, in hindsight, as a list of mechanisms the incident walked straight past.
- The defect that forces every standby copy to rebuild after a promotion tied an instant action to an hours-long consequence. Promotion finished in moments; the rebuild it required ran until 21:20, its length set by the size of the database. Capacity planning for the login database assumed a shape (a primary plus a read-absorbing standby) that the failover itself destroyed. A failover that leaves the survivor unable to carry the load is not a failover; it is a slower outage.
- The backup that saved the API was valuable precisely because it sat outside the automation. The secondary data center's read copies were not in the local failover process, which is why the misfiring automation never touched them, and also why using them needed manual, unautomated traffic steering under pressure. The same property was both the weakness (slow to invoke by hand) and the safeguard (impossible for the automation to break). Automating that steering, which Cloudflare put on its roadmap, would move it from one side of that ledger to the other.
- Under partial failure, the mitigations compete. API calls could cross the ocean; dashboard sessions could not. Improving API availability made the dashboard worse, until the team deliberately sent logins back and accepted the risk of the overloaded primary. There was no single move that helped everything at once, which is the signature of a partial failure: you are choosing which part to protect, not whether to protect all of it.
Patterns in this article
- Generic Mitigation
The two levers Cloudflare pulled, shedding optional load and steering work to untouched capacity, share a useful property: neither requires understanding the failure in order to help. You shed work that can wait to buy the primary headroom, and you move reads to a data center the incident never reached. Cloudflare improvised both levers in the middle of the incident, then put automating the read-steering on its roadmap, so that next time the lever is ready to pull rather than invented under pressure.
- Fault Isolation
The secondary data center's read copies survived because they sat outside the failure's blast radius: not registered in the local promotion process, and not coordinated through the affected etcd cluster. That isolation was partly an accident, since the automation simply had not reached them yet, which sharpens the lesson rather than weakening it. Staying independent of the control plane is itself a fault-isolation boundary, and automating across that boundary spends it.
- Conservative Auto-Remediation
When a fix is expensive or hard to undo, the trigger that fires it should demand evidence proportional to the cost, and time is a form of evidence: six minutes of ambiguity should not launch an hours-long, irreversible cure. Cloudflare's response was not better detection (a partial failure defeats detection by definition) but a slower trigger, tuned per database to how costly the cure is. The faster and cheaper the remedy, the twitchier the trigger can safely be; the slower and more destructive it is, the more it should wait.
Also solving this
Other systems in behindscale's Gray failure defeats automatic detection class: