When the Queue Pushes Back: DoorDash's Escape from RabbitMQ
In mid-2019, the Celery-and-RabbitMQ system running over 900 asynchronous tasks at DoorDash kept going down under load - and when task processing went down, DoorDash effectively went down. Sudden bursts slowed RabbitMQ, and its own defenses made things worse. Flow Control throttled the app servers sending tasks, which felt it as network latency, and that piled requests up upstream. harakiri, a setting that kills any worker running too long, then killed the slow ones, whose restarts flooded the broker with new connections and more load. The high-availability mode meant to help instead cut throughput, its failovers taking 20-plus minutes and losing messages. The escape was a custom Kafka-based system: each task's name and arguments on Kafka, behind a wrapper routing tasks to the old or new system by a feature flag. It shipped in two weeks and cut RabbitMQ load 80% within a week. Then the honest part: the new system had its own new problems - one slow message could stall a whole partition, fixed with a small local queue.
Push a peak burst at RabbitMQ and watch Flow Control throttle the app servers into latency while worker restarts flood the broker with even more load - then restart it and hope. Try the high-availability mode that cuts throughput and jams its own failovers. Then escape to the Kafka MVP, watch 80% of the load leave in a week, and meet the new system's own problems: one slow message freezing a whole partition, until a small local queue makes it freeze just one worker instead.
Problem
RabbitMQ and Celery were mission-critical, powering over 900 asynchronous tasks including order checkout, merchant order transmission, and Dasher location processing - and RabbitMQ was frequently going down under excessive load. When task processing went down, DoorDash effectively went down: orders could not be completed, costing merchants and Dashers revenue and consumers their dinner. The outages clustered at peak demand, and recovery meant restarting the system or standing up an entirely new broker and manually failing over.
The availability trouble has four strands. First, Celery's countdown/ETA feature let engineers schedule tasks into the future, and heavy use parked that future work in the broker. That load increase directly preceded some outages, and was eventually resolved by restricting countdowns in favor of a separate scheduling system. Second, sudden traffic bursts left RabbitMQ in a degraded state where task consumption ran far below expectation, resolvable in their experience only by bouncing the broker. Flow Control - RabbitMQ's mechanism for slowing connections that publish too quickly, so queues can keep up - was often though not always implicated. When it kicked in, publishers experienced it as network latency, which during peak traffic cascaded as requests piled upstream. Third, the uWSGI web workers ran harakiri, killing any process past a timeout. So broker slowness triggered kills, kills triggered reconnection churn across thousands of workers, and churn loaded the broker further: the slowdown feeding itself. And fourth, Celery consumers sometimes simply stopped processing with no load or resource constraint, resuming on a bounce - never root-caused, suspected in the workers rather than the broker.
Scale offered no exit. They were on the largest single-node RabbitMQ available with nothing above it. The primary-secondary HA mode reduced throughput through replication, trading headroom they couldn't spare, while in practice its failovers took more than 20 minutes, often got stuck awaiting manual intervention, and lost messages along the way. Observability was thin on both halves (limited RabbitMQ metrics, opaque Celery workers), operations meant late-night manual failovers, and there were no in-house Celery or RabbitMQ experts to devise a scaling strategy. The engineering time spent keeping it alive was not sustainable.
Solution
DoorDash weighed five options honestly. Swapping Celery's broker to Redis gave no real horizontal scale in the mode they could use, and didn't fix the stuck workers. Swapping to Kafka under Celery wasn't supported. Sharding across multiple brokers spread the load but fixed nothing else - not observability, not churn, not stuck workers. Upgrading versions might help but guaranteed nothing and forced a Python upgrade. The last option was to build a custom Kafka-backed system. It was more work than all the others, but the only one that addressed every failure they'd seen, harakiri churn and stuck consumers included, and they had in-house Kafka expertise to lean on. They chose it, guided by three principles: get something working fast, make it painless for developers to adopt, and roll out gradually with zero downtime.
The MVP was deliberately minimal. Producers put a task's full name and its arguments (serialized with Python's pickle) onto Kafka; consumers look the task up by name and run it. A wrapper around Celery's @task annotation routed each task to the old or new system by a feature flag that could be flipped at runtime. The interface was the same for both, so an adopting team flipped exactly one flag and changed nothing else. To ship sooner, they supported only a whitelist of task settings, the smallest set that covered most tasks. Two weeks of work put it in production; one week of ramp-up, lowest-risk tasks first, cut RabbitMQ load by 80%, and the outages stopped almost as soon as rollout began. The switch had a real cost. Because either system could be chosen per task at runtime, the worker fleet ran at double size, half for each system, taxing enough that a new Kubernetes cluster was spun up just to hold the workers. It wound back down as tasks moved over. Any Celery feature still missing was ranked by how many tasks needed it; the rare ones were never built, and those few tasks were rewritten instead, until everything had moved.
Then the post does what this class demands: it names the new system's own new problems. Kafka hands each partition to one consumer in order, so a single slow message blocks everything behind it in that partition - bad for a high-priority topic. The fix separates fetching from running. One process per worker pulls messages off Kafka into a small local queue (capped at a size you set), and several other processes take work off that queue and run it. Now a slow message ties up just one of those runners while the partition keeps moving, and the cap limits how many in-flight messages a crash could lose. Deployments, which happen several times a day, are the other snag: each one makes Kafka pause briefly to reshuffle which consumer owns which partition. That is fine for a planned release but risky for an emergency hotfix; the wished-for cure is a gentler reshuffling their Kafka client doesn't support yet. The wins were concrete: outages stopped, task processing stopped being the growth ceiling, every queue and worker and task got real metrics, and operations spread out across teams through templated alerts and clear per-topic owners. The conclusion names its own rule of thumb: 80% of the result for 20% of the effort, a quick fix that buys time for the fuller one.
Tradeoffs
- A buffer's own defenses can be what carries its trouble outward. Flow Control exists so queues can keep up, but its throttling reaches the app servers as unexplained network latency, which at peak piles their requests up. A buffer that responds to overload by slowing its senders hasn't absorbed the pressure; it has passed it, invisibly, to the layer least able to figure out what happened.
- Timeout-and-kill loops can amplify the very thing they're meant to contain. harakiri kills any worker that runs past its timeout, which is sensible for one worker but dangerous for a whole fleet. Broker slowness kills workers, thousands of them restart at once, and those restarts open new connections. Reconnections are themselves a known source of RabbitMQ load, so the slowdown deepens. Any fix that responds to slowness by creating reconnection work should be checked for this loop before an incident, not during one.
- High availability that costs throughput can end up delivering neither. Keeping a synced backup copy of the broker cut throughput - headroom they couldn't spare - and the takeover, when tested, took more than 20 minutes, got stuck needing manual help, and lost messages. A backup is only worth its speed cost if the takeover actually works. Theirs turned 'we might go down' into 'we go down slower, with less headroom, and lose messages coming back.'
- Whether to repair or replace turned on what was actually reachable, not on preference. Weighing the five options, each repair failed on a specific gap: Redis couldn't scale out in a usable way, sharding the broker only spread the load, version upgrades guaranteed nothing. The custom Kafka build won as the only option that covered every failure they'd seen, including the stuck consumers they never fully explained. With no in-house RabbitMQ expertise and a system they couldn't see into, fixing the old broker wasn't reachable from where they stood, so replacing it was the move.
- Migration safety has a price tag, and they paid it in the open. Being able to switch each task between the two systems at runtime meant running the worker fleet at double size, half per system, with a whole new Kubernetes cluster to hold them. That was the insurance premium for maintenance, load shedding, and instant rollback. The matching triage: rank every missing Celery feature by how many tasks used it, build the common ones, rewrite the tasks that used the rare ones. Two weeks to production, 80% of the load gone a week later - the same 80/20 logic applied to both the feature set and the migration.
- The new system came with its own new problems, named as plainly as the old ones. Kafka delivers each partition in order, so one slow message stalls its whole partition. The fix separates fetching from running, through a small capped local queue: one runner stalls, the partition keeps moving, and the cap limits what a crash can lose. Deploys still cause brief stalls while Kafka reshuffles its work, awaiting a gentler reshuffle their client doesn't support yet. You can't escape queue trouble; the win is choosing trouble you can see, bound, and live with.
Patterns in this article
- Fetch-Execute Decoupling
This pattern comes straight from the head-of-line fix: one process per worker pulls messages off Kafka into a small capped local queue, and several other processes take work off that queue and run it. So a slow message ties up just one runner while the partition keeps moving, and the cap limits how many in-flight messages a crash can lose. It sits right next to Selective Acknowledgment, the shape Uber used for the same problem. Uber's proxy keeps its delivery guarantees with out-of-order acknowledgments and a dead-letter queue; DoorDash's local queue accepts a small window of possible loss for simplicity. Two different prices for unblocking the same slow-message problem.
- Universal Staged Rollout
Roll a big migration out gradually behind a switch, rather than all at once. Feature flags at both the sending and consuming ends let DoorDash flip any task between the old and new systems at runtime. That bought instant rollback and the ability to move cluster by cluster, paid for by running the worker fleet at double size, a new Kubernetes cluster included. Missing features were ranked by how many tasks used them: the common ones were built, the rare ones' tasks rewritten. Two weeks to a working version in production, lowest-risk tasks first, 80% of the load moved within a week of launch.
Also solving this
Other systems in behindscale's Buffer degrades under backlog class: