Pattern · seen in 3 breakdowns across 3 companies

Circuit Breaker

A circuit breaker watches the calls to a dependency and, once too many fail, stops sending calls and fails them instantly for a while, so a down dependency can't tie up the whole system.

resilience

The mechanism

The pattern at its core: a stream of calls to a dependency that goes down for a while, and a breaker that steps in so those calls fail instantly instead of each waiting out the full timeout.

LIVE ARTIFACTTHE PATTERN, GENERALOPEN FULL SCREEN ↗
THE IDEAWhile a dependency is down, a call with only a timeout still waits the full timeout before failing - and every waiting call holds a thread the system needs. A breaker notices the run of failures, trips open, and fails the next calls instantly instead of waiting. That frees the system and takes load off the dependency so it can recover.
WHAT TO TRYSend a burst of calls through an outage with timeouts only, then with a breaker. With only timeouts, every call waits the full time; the breaker trips after a few failures and the rest fail instantly, so far less time is wasted waiting.

Send calls through an outage with timeouts only, then with a breaker - watch the wasted waiting time collapse.

Definition

A circuit breaker wraps the calls your service makes to something it depends on - a database, a payment provider, another service - and monitors those calls. It has three states:

  • closed - the normal, healthy state; calls flow straight through (similar to a closed electrical circuit)
  • open - state reached after too many recent failures; calls fail instantly, without even trying the dependency
  • half-open - state reached after a short cooling-off wait period; one test call is let through to see if the dependency is back

It starts in closed state and switches to open state when failures pile up. After a short cooling-off wait period, it switches to half-open state. From that state, a successful test call closes it again; a failed one opens it back up for another wait.

THE BREAKER'S THREE STATES
A state diagram: closed, open, and half-open, with arrows for tripping open, cooling down, and probing to recover.
The breaker starts closed, letting calls through. Too many failures trip it open, so calls fail instantly. After a cooldown it goes half-open and sends one test call: success closes it again, a failure opens it back up.

Why not just rely on a timeout? Because a timeout is not free. It caps how long one call waits, but while the dependency is down, every new call still waits out the full timeout before giving up - tying up threads, connections, and queue slots at the worst possible moment. An outage usually stays down for a while, rather than clearing on its own within seconds. So instead of every call waiting out the full timeout for the whole outage, the breaker makes them fail right away. Pulling traffic away also gives the struggling dependency room to recover. Failing instantly makes room for a planned fallback - a cached value, a simpler path, a clear error message - but that fallback is work your application has to do; the breaker only creates the opening.

The main design decision is what exactly each breaker protects. Make its scope too broad and one small part failing trips the breaker for calls that would have worked fine. Make it too narrow and you end up with a breaker for everything, each one slower to notice trouble and each needing its own tuning. Nearly every ecosystem ships a circuit-breaker library, so writing the code is rarely the hard part. The common problem is a breaker added in a hurry: its trip thresholds get set once and never looked at again, and no one has decided what should actually happen when it trips.

When it applies

01The dependency tends to fail in bursts. When it breaks, many calls fail together in a short window rather than one unlucky call here and there.
02Waiting for the timeout costs more than failing fast. Every call stuck waiting holds a thread, a connection, or a queue slot that the system needs back.
03You have a fallback, or a fast error beats a slow one. A cached value, a simpler path, or a clear error returned instantly is better than making the caller wait.
04The dependency recovers faster when you leave it alone. Pulling traffic off a struggling service can be what lets it catch up and come back.

Tradeoffs

The breaker only makes the opening. The fallback is still the application's job: deciding what to do with that instant failure - a cached value, a degraded path, a clear error - is work you have to design.
Bad thresholds or scope can do real damage. Set them wrong and a breaker can trip healthy traffic, or waste a lot of resources at scale before anyone notices.
How narrowly to scope each breaker is a standing design problem. Scope it broadly and one part failing trips calls that were fine; scope it finely and there is far more breaker state to tune.
Fast failures skew your latency graphs. A flood of instant failures looks fast, so success and failure have to be graphed separately or the dashboards lie.

The same move, 3 ways

Every row is a production system that bet on this pattern — the note says how, in that system's own terms.

DoorDash
DoorDash Engineering Blog
2023
Third company, and the first appearance as antagonist. Shopify aims breakers at payment gateways, Meta at its own MySQL shards; DoorDash builds them into the internal gRPC client and then documents a misconfigured breaker turning a database slowdown into a wide-blast-radius outage. The recurrence adds the missing caveat: a breaker is a local actor whose threshold encodes a guess about the whole system that the whole system never confirms. Read the breakdown →
Shopify
Shopify Engineering
2022
Semian wraps Net::HTTP, MySQL, Redis, and gRPC. The notable refinement is identifier granularity: breaker state is scoped to endpoint plus merchant country code, so an outage at one country's local acquirer trips only that country's circuit. Read the breakdown →
Meta
Engineering at Meta
2021
The post names the circuit breaker directly. Each MySQL shard's enqueue worker watches slow-query and error rates over rolling windows, marks the shard down, and stops feeding it new items until it recovers, refusing to deepen an overload already in progress. It is the same reflex as Shopify's payment-gateway breakers, but pointed one layer down the stack - at your own storage, not an outside partner. Read the breakdown →

Problems this pattern answers

The walls where its breakdowns live — each opens the cross-company comparison.

Ambiguous failure under retrySHOPIFY →Buffer degrades under backlogMETA →Mitigation scoped narrower than the failureDOORDASH →