When the Queue Pushes Back: DoorDash's Escape from RabbitMQ

RabbitMQ and Celery were mission-critical, powering over 900 asynchronous tasks including order checkout, merchant order transmission, and Dasher location processing — and RabbitMQ was frequently going down under excessive load. When task processing went down, DoorDash effectively went down: orders could not be completed, costing merchants and Dashers revenue and consumers their dinner. The outages clustered at peak demand, and recovery meant restarting the system or standing up an entirely new broker and manually failing over.

The availability anatomy has four strands. Celery's countdown/ETA feature let engineers schedule tasks into the future, and heavy use parked that future work in the broker — load increases that directly preceded some outages, eventually resolved by restricting countdowns in favor of a separate scheduling system. Sudden traffic bursts left RabbitMQ in a degraded state where task consumption ran significantly below expectation, resolvable in their experience only by bouncing the broker; Flow Control — RabbitMQ's mechanism for slowing connections that publish too quickly so queues can keep up — was often though not always implicated, and when it kicked in, publishers experienced it as network latency, which during peak traffic cascaded as requests piled upstream. The uWSGI web workers ran harakiri, killing any process exceeding a timeout — so broker slowness triggered kills, kills triggered reconnection churn across thousands of workers, and churn loaded the broker further: the slowdown feeding itself. And separately, Celery consumers sometimes simply stopped processing absent any load or resource constraint, resuming on a bounce — never root-caused, suspected in the workers rather than the broker.

Scale offered no exit. They were on the largest single-node RabbitMQ available with nothing above it, and the primary-secondary HA mode reduced throughput through replication — trading headroom they couldn't spare — while in practice failovers took more than 20 minutes, often got stuck awaiting manual intervention, and lost messages along the way. Observability was thin on both halves (limited RabbitMQ metrics, opaque Celery workers), operations meant late-night manual failovers, and there were no in-house Celery or RabbitMQ experts to devise a scaling strategy. Engineering time spent keeping it alive was not sustainable.

The options table is the decision's honest record: swap Celery's broker to Redis (no horizontal scale in the mode they could use; doesn't fix stuck workers), swap to Kafka under Celery (unsupported), shard across multiple brokers (dilutes load, fixes nothing else — not observability, not churn, not stuck workers), upgrade versions (might help, guarantees nothing, demands a Python upgrade), or build a custom Kafka-backed system — more work than every other option, and the only one addressing every observed failure, harakiri churn and stuck consumers included, with in-house Kafka expertise to lean on. They chose the last, governed by three principles: hit the ground running (they liken it to swapping a fuel pump while driving the race car), design for seamless developer adoption, and roll out incrementally with zero downtime.

The MVP was deliberately minimal: producers publish a task's fully qualified name and pickled arguments to Kafka; consumers import the task by name and execute it synchronously. A wrapper around Celery's @task annotation routed each submission to either system via dynamically configurable feature flags — the same interface for both worlds, so adopting teams implemented exactly one flag and nothing else. A whitelist of compatible task parameters kept scope to the smallest feature set covering a majority of tasks. Two weeks of development put it in production; one week of ramp-up — lowest-risk, lowest-priority tasks first — cut RabbitMQ task load by 80%, and the repeated outages stopped almost as soon as rollout began. The migration machinery was priced candidly: dynamic flags at both submission and consumption sides meant the worker fleet ran at double capacity, half devoted to each system — taxing enough that a new Kubernetes cluster was spun up just to house workers — then wound down as topics transitioned and confidence grew. Remaining Celery features were ranked by the number of tasks using them; features used by few tasks were never implemented, those tasks rewritten instead, until everything had moved.

Then the post does what the class demands: it documents the new buffer's own backlog physics. Kafka partitions deliver in order to one consumer per group, so a single slow message stalls everything behind it in its partition — disastrous for a high-priority topic. The fix decouples fetching from execution: one Kafka-consumer process per worker feeds a local queue, bounded by a user-defined threshold, from which multiple task-execution processes draw — a slow message now stalls one executor while the partition keeps flowing, with the threshold capping the in-flight messages that a crash could lose. And deployments — several per day — trigger partition rebalancing that momentarily halts consumption despite per-topic consumer groups; tolerable for planned releases, potentially cascading for an emergency hotfix, with incremental cooperative rebalancing the desired cure once their client supports it. The wins ledger: outages stopped, task processing no longer the growth ceiling, granular metrics baked into every queue, worker, and task, and operations decentralized — templatized Terraform alerts, explicit owners for every topic and implicitly all 900+ tasks, day-to-day operations self-serve. The conclusion names its own doctrine: 80% of the result with 20% of the effort, a tactical mitigation buying time for the strategic solution.