Pattern · seen in 6 breakdowns across 6 companies

Universal Staged Rollout

Universal staged rollout means every change - not just code, but config and auto-updates - rolls out gradually, so a bad one is caught early instead of hitting all the application servers at once.

resilience

The mechanism

The pattern at its core: a fleet of servers, one change to ship, and two ways to ship it - all at once, or a small wave at a time with a health check between waves.

LIVE ARTIFACTTHE PATTERN, GENERALOPEN FULL SCREEN ↗
THE IDEAA bad change caught in a small first wave is a small problem. The same change pushed to every server at once is an outage. Staged rollout is just insisting on the first path - for every kind of change, not only deploys.
WHAT TO TRYShip a bad change all at once and watch it take every server. Then stage it: deploy to the canary, read the health signal, and roll back before it spreads.

Send a bad change through the channel nobody staged - it takes the whole fleet at once.

Definition

Every production change has to go out the same careful way: a little at a time, monitored, then widened. The real danger is the change path nobody thinks of as change - an auto-update that is on by default, a vendor tool that upgrades itself, a scheduled job that runs on every machine at the same minute. A path like that turns one hidden bug into an outage that hits every machine at the same moment, and it does so precisely because nobody ever listed it as a way change gets in.

So list every way a machine can change, and either put each one through staged rollout or turn it off. The paths that count:

  • code deploys
  • configuration changes
  • infrastructure automation
  • OS and security updates
  • vendor agents that upgrade themselves
  • scheduled jobs that run everywhere at once
THE CHANNEL NOBODY STAGED
Several change channels pass through a staging gate to the fleet; one auto-update channel bypasses the gate to every node.
Most ways a change reaches the fleet run through the same staging gate - small first, watched, then widened. The danger is the channel that skips the gate, like an auto-update on by default, and reaches every machine at once.

Shutting a fast channel off is cheapest when it is redundant - when something else already delivers the same change through a staged path, closing the fast one costs nothing. When it is not redundant, the pattern puts the real tradeoff on the table: how much change the fleet can safely take at the same time, against how fast a fix needs to reach everywhere. This is different from Conservative Auto-Remediation: that pattern limits how eagerly automation reacts to failures; this one limits how much change of any kind reaches the fleet at once.

When it applies

01You have auto-updates turned on by default: OS or vendor tools that upgrade themselves on every machine, and nobody has checked whether they go out gradually or all at once.
02Every machine is basically identical: one OS version, one base image, so a single bad change reaches the whole fleet by construction, with nothing to stop it partway.
03You are hardening after an outage: if the trigger hit everywhere at the same moment, that is a sign the change went out to every machine at once, unstaged.

Tradeoffs

Staging slows down security patches: rolling a fix out gradually means some machines stay unpatched longer, so you are trading a smaller blast radius for a slower patch - unless a safe fast path already exists.
The audit is never finished: every new vendor agent, tool, or piece of automation is another possible change channel you have to find and bring under staging.
It leaves you running mixed versions: while a change is only part way across the fleet, different machines run different versions, which is harder to debug and support - the price of capping correlated failure.

The same move, 6 ways

Every row is a production system that bet on this pattern — the note says how, in that system's own terms.

Datadog
Datadog Engineering Blog
2023
This one comes from the fix, not the failure. After the incident, Datadog turned off the old automatic-update channel, at no cost to security, because the same updates already arrive through a staged, controlled process, and audited the fleet for other channels that skip that discipline. The claim: every channel that can change the fleet, including operating-system security updates, should go through a staged rollout, because an unstaged one turns any hidden flaw into a single synchronized global failure. Read the breakdown →
Reddit
r/RedditEng
2023
Recurs in the direction nobody practices: readmission. Traffic returned at 1% and walked up through eight steps to 100%, with idle-scaled services and cold caches treated as what they are — a system that cannot absorb normal load yet — and touchy legacy services hand-gated back in. Staged rollout applied to recovery load rather than to a migration. Read the breakdown →
Canva
Canva Engineering Blog
2022
The migration end to end: a dual-read comparison in production until replication bugs were fixed, slightly-stale reads with a fallback for stragglers, and test matrices on both implementations for the write cutover. A rehearsed run book and a flag that priced rollback at seconds carried it through development and staging before production. It is another migration in this collection where nothing cut over on faith, the whole risk retired in stages before the irreversible step. Read the breakdown →
GitLab
GitLab Blog (Engineering)
2022
The rollout's sharpest move: make the app behave as if it already had two databases (two full sets of connections) while both still point at one real database. That turns a single hard migration into seven phases, the first six easy to undo and safe to ship early, leaving the real switch as a tiny change to one host. Seven full practice runs against a two-hour budget brought the real run in at 93 minutes. Read the breakdown →
Slack
Slack Engineering
2020
The three-year migration is this pattern at datastore scale. A deliberately small first use case in production (RSS feeds into channels) forced every operational surface to exist early. Then tables were cut over one at a time under a generic backfill, with the application writing to both stores and a parallel system comparing reads to prove the two matched before any traffic committed. The result went from 0% to 99%, with the last percent scheduled, and a 50%-in-one-week COVID surge was absorbed mid-journey. Nothing cut over on faith, ever. Read the breakdown →
DoorDash
DoorDash Engineering Blog
2020
Roll a big migration out gradually behind a switch, rather than all at once. Feature flags at both the sending and consuming ends let DoorDash flip any task between the old and new systems at runtime. That bought instant rollback and the ability to move cluster by cluster, paid for by running the worker fleet at double size, a new Kubernetes cluster included. Missing features were ranked by how many tasks used them: the common ones were built, the rare ones' tasks rewritten. Two weeks to a working version in production, lowest-risk tasks first, 80% of the load moved within a week of launch. Read the breakdown →

Problems this pattern answers

The walls where its breakdowns live — each opens the cross-company comparison.

Single-cluster scaling ceilingGITLAB · SLACK →Buffer degrades under backlogDOORDASH →Observer shares fate with observedDATADOG →Single-table scaling ceilingCANVA →Unrecorded config outlives its authorsREDDIT →