The Queue That Couldn't Drain: Kafka in Front of Redis at Slack

Slack's job queue is the asynchronous half of the product: business logic too time-consuming for a web request — every message post, push notification, URL unfurl, calendar reminder, and billing calculation — flows through it, over 1.4 billion jobs on the busiest days at a peak of 33,000 per second. The original architecture was the classic Redis task queue, essentially unchanged through orders of magnitude of growth: the web app hashes a job identifier to one of the Redis hosts, deduplicates against jobs already in queue, and pools of workers poll for work, moving each job to an in-flight list while executing and to a retry queue on failure.

The outage that forced the re-evaluation began outside the queue: resource contention in the database layer slowed job execution. Workers dequeued slower than the web app enqueued, and Redis climbed to its configured memory limit. At that point no new jobs could be enqueued — every Slack operation that depends on the queue began failing. The deeper trap was on the drain side: the system required a bit of free Redis memory in order to dequeue a job, because dequeuing moves the job into a processing list. A full Redis therefore could not empty itself. Even when the database contention was resolved, the job queue remained locked, and recovery took extensive manual intervention.

The post-mortem concluded that scaling the existing system was untenable, and named the constraints plainly. Redis had little operational headroom, particularly memory — sustained enqueue-over-dequeue ends in exactly the deadlock above. Dequeue cost was proportional to queue length, thanks to earlier data-structure choices, so longer queues were harder to empty — an unfortunate feedback loop in the post's own words. Workers couldn't scale independently of Redis: every added worker added polling load, so attempting to increase execution capacity could overwhelm an already overloaded Redis — a second feedback loop. Client connections formed a complete bipartite graph, every enqueuer needing current knowledge of every Redis instance. And the queue's delivery semantics were unclear enough that engineers were reluctant to lean on a system that was already fundamental to the architecture.

The team weighed a ground-up rewrite against incremental change and chose the smallest change that removed the existential failure mode: put Kafka in front of Redis, rather than replace Redis outright. Replacing Redis would have meant rewriting the scheduling, execution, and deduplication logic riding on it; adding a durable buffer in front left the application's enqueue and dequeue interfaces intact while ending the era of backlog living in the memory the drain depends on.

Two stateless Go services bracket the new tier. Kafkagate accepts a simple HTTP POST from the PHP/Hacklang web app and relays it into Kafka — persistent broker connections, leadership-change tracking, deliberately minimal client semantics. Its design is explicit about its bets: it waits only for the Kafka leader's acknowledgment, not replication, trading a small risk of loss on a broker crash for the lowest possible enqueue latency — a bias toward availability over consistency the post judges right for most of Slack's semantics, with stronger guarantees contemplated as an option for critical jobs. Synchronous writes give the web app a positive acknowledgment or an error, tightening semantics engineers had found hard to trust. Requests route preferentially to Kafkagate instances in the same availability zone, cutting latency and transfer cost while retaining cross-AZ failover.

JQRelay drains Kafka into Redis, and it is where the queue's new discipline lives. Each relay instance acquires a Consul lock for one Kafka topic — an EC2 auto-scaling group plus the lock protocol guarantees exactly one relay per topic with self-healing on failure. It advances Kafka's commit offset only after a job is successfully written to Redis, retrying indefinitely through Redis outages; job-specific errors are re-enqueued to Kafka rather than silently dropped, so a poison job can't block a queue and can't vanish either. Crucially, JQRelay writes to Redis under rate limits configured in Consul — the admission valve that keeps Redis inside its memory headroom no matter how fast the world enqueues. The Kafka tier itself: 16 brokers on i3.2xlarge, 50 topics of 32 partitions each, replication factor 3, two-day retention, rack-aware across availability zones.

The rollout matched the caution of the design. Double-writes sent every job to both systems while JQRelay ran in shadow mode, exercising the full new path on production traffic without executing anything. Correctness was established by counting jobs at every hop — web app to Kafkagate, Kafkagate to Kafka, Kafka to Redis — and by heartbeat canaries enqueued every minute to each of the 1,600 partitions, alerting on end-to-end flow and timing. Failure testing killed brokers singly, in pairs within an AZ, and all three replicas at once to force unclean leader election. Then Slack ran the system on itself for weeks before rolling it out job type by job type. The conclusion rewrites the outage: a queue build-up now lands in durable storage while enqueues keep succeeding, and the response is turning JQRelay's rate limits — not paging humans to unstick a seized Redis.