The Queue That Couldn't Drain: Kafka in Front of Redis at Slack

Slack's job queue is the asynchronous half of the product: work too time-consuming to finish inside a web request - every message post, push notification, link unfurl, calendar reminder, and billing calculation - flows through it, over 1.4 billion jobs on the busiest days at a peak of 33,000 per second. The original design was the classic Redis task queue, essentially unchanged through orders of magnitude of growth: the web app hashes a job's identifier to one of the Redis hosts, checks it against jobs already queued to avoid duplicates, and pools of workers poll for work, moving each job to an in-flight list while it runs and to a retry queue if it fails.

The outage that forced a rethink began outside the queue: resource contention in the database layer slowed job execution. Workers now drained slower than the web app enqueued, and Redis climbed to its configured memory limit. At that point no new jobs could be enqueued, and every Slack operation that depends on the queue began failing. The deeper trap was on the drain side: dequeuing a job moves it into a processing list, which needs a little free memory. A full Redis therefore could not empty itself. Even once the database contention was resolved, the queue stayed locked, and recovery took extensive manual work.

WHY A FULL QUEUE SEIZES

The post-mortem concluded that scaling the existing system was untenable, and named the constraints plainly. Redis had little room to spare, especially in memory, so a sustained period of enqueuing faster than dequeuing ends in exactly the deadlock above. Dequeue cost was proportional to queue length, thanks to earlier data-structure choices, so longer queues were harder to empty - the post's own "unfortunate feedback loop." Workers couldn't scale independently of Redis: every added worker added polling load, so trying to add execution capacity could overwhelm an already struggling Redis (a second feedback loop). Every enqueuer also had to track every Redis instance, a full mesh of connections that grew on both sides at once. And the queue's delivery guarantees were vague enough that engineers were reluctant to lean on a system already fundamental to the architecture.

The team weighed a ground-up rewrite against an incremental change and chose the smallest change that removed the existential failure mode: put Kafka in front of Redis instead of replacing Redis outright. Replacing Redis would have meant rewriting the scheduling, execution, and deduplication logic built on it; adding a durable buffer in front left the application's enqueue and dequeue interfaces intact while ending the era of backlog living in the very memory the drain depends on.

Two stateless Go services sit on either side of the new tier. The gateway, Kafkagate, takes a simple HTTP POST from the PHP/Hacklang web app and writes it into Kafka. Its design states its bets plainly. Kafka copies each message to several brokers for safety; Kafkagate waits only for the lead broker to acknowledge the write, not for the copies, which trades a small risk of loss if that broker dies at the wrong moment for the lowest possible enqueue latency. The post judges that availability-over-consistency bias right for most of Slack's jobs, with stronger guarantees left open as an option for critical ones. The write is synchronous, so the web app gets back a clear success or error, tightening a guarantee engineers had found hard to trust. To cut latency and cost, requests prefer a Kafkagate instance in the same availability zone, while still failing over across zones when needed.

The relay, JQRelay, drains Kafka into Redis, and it is where the queue's new discipline lives. Each relay instance takes a lock, held in Consul (a coordination service the fleet uses to agree on who holds what), for one Kafka topic. That lock plus an auto-scaling group guarantees exactly one relay per topic, healing itself if one fails. It advances Kafka's commit offset only after a job is safely written to Redis, retrying indefinitely through Redis outages, and re-enqueues a failed job back to Kafka rather than dropping it, so a job that keeps failing can neither block the queue nor vanish. Crucially, JQRelay writes to Redis under rate limits configured in Consul: this is the admission valve that keeps Redis inside its memory headroom no matter how fast the world enqueues. The Kafka tier itself: 16 brokers on i3.2xlarge, 50 topics of 32 partitions each, replication factor 3, two-day retention, spread across availability zones.

The rollout matched the caution of the design. Double-writes sent every job to both systems while JQRelay ran in shadow mode, exercising the full new path on real traffic without executing anything. Correctness was checked by counting jobs at every hop - web app to Kafkagate, Kafkagate to Kafka, Kafka to Redis - and by heartbeat jobs enqueued every minute into each of the 1,600 partitions, alerting on end-to-end flow and timing. Failure testing killed brokers one at a time, in pairs within a zone, and all three copies at once to force Kafka to promote a broker that might be missing recent writes. Then Slack ran the system on itself for weeks before rolling it out one job type at a time. The result rewrites the outage: a build-up now lands in durable storage while enqueues keep succeeding, and the fix is turning a rate-limit dial rather than paging humans to unstick a seized Redis.