Neither Dead nor Alive: Cloudflare's Real-World Byzantine Failure
Cloudflare's control plane - the part that configures and manages the service (its API and dashboard), not the network that carries customer traffic - degraded for six hours and 33 minutes on November 2, 2020, with API success dipping to 75% and the dashboard up to 80 times slower. The trigger lasted six minutes: one network switch entered a partially operating state, healthy enough that failover never fired, broken enough that one node of etcd (the store the control plane relies on to keep every copy of its coordination data in agreement) lost contact with its leader while every other path stayed up. Three etcd nodes held three views of reality, tied elections made the cluster read-only, and the database cluster manager read the silence as a failed primary and auto-promoted a replica. A known defect then forced a rebuild of every replica, leaving a read-heavy authentication database on a lone, overwhelmed primary for hours. Every layer had redundancy; every layer's redundancy assumed a component is either fully up or fully down.
Kill the switch dead and watch failover shrug — then set it half-alive, watch three etcd nodes hold three realities, and decide how fast the promotion trigger fires. Mitigate by hand, or set the trigger past six minutes and meet the outage that didn't happen.
Problem
Cloudflare's control plane runs as microservices redundant across two regions, and the post enumerates its redundancy discipline deliberately: paired active-active switches per rack, either able to carry the load alone; critical services spread across racks; dual power feeds; RAID 10 or three-machine replication for anything critical. Redundancy at each layer is reviewed and required. The post opens with the question that discipline invites: so how could things go wrong?
At 14:43 UTC the switch answered. Two of its protocols stayed alive: LACP, which bundles several physical links so they act as one and spreads traffic across them, and BGP, which announces to the network which routes the switch can carry. A third, vPC, the protocol that makes a pair of switches look like one, went down, and the switch's forwarding hardware quietly stopped processing some fraction of the packets it received. This is the failure mode redundancy audits don't model: the audit asks 'what if this device dies?', and the paired switch is the answer. Nothing asks 'what if it keeps claiming to be alive while doing a third of its job?', because the failover trigger is the device dying. Six minutes later the switch recovered on its own. The impact was just beginning.
The rack held one member of the etcd cluster Cloudflare's core data centers use for coordination. Because LACP was spreading each server's traffic across links, the damage landed on exactly one path: the link from node 1 to node 3 (node 3 was the cluster's leader at the time). Node 1, unable to reach the leader, repeatedly called elections and voted for itself; node 2, which could still see node 3, kept voting for node 3. Ties resolved nothing node 1 could reach, and because RAFT elections block writes until they settle, the cluster went read-only for the switch's six-minute half-life. Then the chain fired: with etcd read-only, two database clusters could no longer confirm they had a healthy primary. The automation drew the crash-stop conclusion, that no signal means a dead primary, and promoted each cluster's synchronous replica (the standby copy kept exactly in step with the primary) to take over - even though the real primary was fine. A synchronous replica stays exactly current, as opposed to the asynchronous replicas that lag slightly behind.
Solution
The promotion itself worked flawlessly, without error or data loss. But a known defect in cluster management required every replica to be rebuilt after a promotion, with rebuild time proportional to database size. One cluster shrugged it off. The other was the authentication database serving API calls and dashboard logins: read-heavy, and dependent on a replica deliberately absorbing reads to spare the primary. Promotion destroyed that arrangement; the new primary took the entire load alone, hours after the switch was already healthy again.
The response was manual, and its shape is the lesson: every effective action was a human working around the automation. First, shed load. The team dialed back SSL certificate pushes, emails, and other deferrable work touching the primary, buying headroom for the hours the rebuild would take regardless. Second, use a redundancy the automation didn't know about: every cluster keeps live replicas in the secondary data center, and because those replicas aren't part of the local failover process, the misfiring automation never touched them. Steering reads to them wasn't automated, so the team manually diverted the API traffic that could tolerate it, substantially improving availability. The unautomated redundancy was the redundancy that survived. Third, accept a tradeoff they couldn't avoid: dashboard sessions live in a Redis store in the primary data center and can't be served from the other region, so the same actions that helped the API were making the dashboard worse. After a period of increasingly frustrated users, the team failed authentication back to the primary region and kept the primary database running as best they could until the rebuild finished. At 21:20 UTC the first replica finished rebuilding and performance returned instantly.
The postmortem's own observation is the thesis: each system had redundancy, and no system fully failed. Each entered a degraded state, and that combination is considerably harder to model. The transferable fix was to revisit how aggressive the auto-remediation should be. Fast promotion had been a point of pride, built because manual promotion once took painfully long, but the incident inverted the pride: for at least one database, the cure may be worse than the disease, and the trigger was retuned immediately. The post closes by defending RAFT (the relatively simple agreement algorithm etcd uses) over full Byzantine fault tolerance, or BFT (a far more complex family of algorithms built to keep working even when some nodes actively lie or send conflicting information). Most cluster management skips BFT because a simple protocol with a rare known hole is safer than a complex one that is hard to implement correctly. And in a note added after publication, Cloudflare appended the community's correction: this was an omission fault (a node that just goes silent on one path), not a true Byzantine fault (a node sending actively conflicting information), and omission faults can be tolerated without going all the way to BFT. The title overclaimed; the postscript said so.
Tradeoffs
- Choosing RAFT over Byzantine fault tolerance is a deliberate trade of safety-against-a-rare-fault for simplicity, and Cloudflare re-signs it even after being burned. A crash-stop consensus algorithm like RAFT is easier to implement, verify, and debug; the price is a known vulnerability to exactly the conflicting-views scenario that occurred. The post defends the position after the incident (a simple protocol with a rare known hole beats a complex protocol you can't trust yourself to implement), and the postscript strengthens it: the actual fault (an omission, one node silently unreachable) is tolerable without going all the way to BFT.
- Fast auto-remediation trades the quality of the evidence for the speed of the reaction. Automatic primary promotion on a minute's notice was built deliberately, because slow manual promotion had hurt before. The same hair trigger, pointed at six minutes of ambiguous silence, executed a promotion nothing needed, and that promotion, through the rebuild defect, was the entire outage. Cloudflare's fix wasn't to remove the automation but to slow it, so the evidence has time to resolve. How fast to apply a cure is a setting, and its safe value depends on how expensive the cure is.
- Redundancy audits assume failure is all-or-nothing, and every layer's answer assumed that same question. Paired switches answer 'what if the switch dies'; RAFT answers 'what if the leader dies'; auto-promotion answers 'what if the primary dies.' A component that half-dies falls through all three answers at once, and because LACP spreads each server's traffic across the bundled links, the packet loss was spread thin enough that no single node ever saw a fully dead link. The post's redundancy inventory reads, in hindsight, as a list of mechanisms the incident walked straight past.
- The defect that forces every replica to rebuild after a promotion tied an instant action to an hours-long consequence. Promotion completed in moments; the replica rebuild it required ran until 21:20, its length proportional to database size. Capacity planning for the auth database assumed a shape (a primary plus a read-absorbing replica) that the failover itself destroyed. A failover that leaves the survivor unable to carry the load isn't a failover; it's a slower outage.
- The redundancy that saved the API was valuable precisely because it sat outside the automation. The secondary data center's read replicas weren't in the local failover process, which is why the misfiring automation never touched them, and also why using them needed manual, unautomated traffic steering under pressure. The same property was both the weakness (slow to invoke by hand) and the safeguard (impossible for the automation to break). Automating that steering, which the post implies is on the roadmap, would move it from one side of that ledger to the other.
- Under partial failure, the mitigations compete. API calls could cross the ocean; dashboard sessions couldn't. Improving API availability made the dashboard worse, until the team consciously failed authentication back and accepted the degraded-primary risk. There was no single move that helped everything at once, which is the signature of a partial failure: you are choosing which part to protect, not whether to protect all of it.
Patterns in this article
- Generic Mitigation
The two levers Cloudflare pulled, shedding discretionary load and steering work to untouched capacity, both share a useful property: they don't require understanding the failure to help. You shed deferrable work to buy the primary headroom, and you move reads to a data center the incident never reached. Slack had pre-built its version of this as a ready button it could press on day one; Cloudflare improvised both levers mid-incident and then made automating the read-steering part of its roadmap, which is the same move as pre-positioning the button.
- Fault Isolation
The secondary data center's read replicas survived because they sat outside the failure's blast radius: not registered in the local promotion process, and not coordinated through the affected etcd cluster. That isolation was partly an accident (the automation simply hadn't reached them yet), which sharpens the lesson rather than weakening it: staying independent of the control plane is itself a fault-isolation boundary, and automating across that boundary is spending it.
- Conservative Auto-Remediation
When a fix is expensive or hard to undo, the trigger that fires it should demand evidence proportional to the cost, and time is a form of evidence: six minutes of ambiguity should not launch an hours-long, irreversible cure. Cloudflare's response wasn't better detection (a partial failure defeats detection by definition) but a slower trigger, tuned per-database to how costly the cure is. The faster and cheaper the remedy, the twitchier the trigger can safely be; the slower and more destructive, the more it should wait.
Also solving this
Other systems in behindscale's Gray failure defeats automatic detection class: