Problem · seen at 5 companies
Buffer degrades under backlog
The queue you added to protect your system becomes the thing that takes it down.
The wall
A buffer that stops absorbing exactly when the backlog it exists for arrives — whether the buffer's substrate degrades under the backlog, or its access semantics let the backlog capture the buffer. 5 teams hit this wall; the breakdowns below are the evidence.
Same wall, 5 systems
Uber
Uber Engineering
Kafka's per-partition ordering lets one stuck message stall every message behind it. Read the breakdown →
Meta
Engineering at Meta
The queue's own substrate slows down exactly as the backlog it must absorb grows. Read the breakdown →
DoorDash
DoorDash Engineering Blog
Under peak bursts RabbitMQ throttled its publishers, and the slowdown cascaded upstream. Read the breakdown →
Segment
Segment Blog
Hundreds of destinations fail daily; a shared queue makes every failure everyone's backlog. Read the breakdown →
Slack
Slack Engineering
A full Redis queue couldn't dequeue, because draining required the very memory the backlog had consumed. Read the breakdown →
Patterns in this class
Every breakdown
Uber Engineering · Aug 31, 2021The Ledger Above the Log: Uber's Kafka Consumer ProxyEngineering at Meta · Feb 22, 2021Delay, Not Loss: FOQS, Meta's Trillion-Item Priority QueueDoorDash Engineering Blog · Sep 3, 2020When the Queue Pushes Back: DoorDash's Escape from RabbitMQSegment Blog · May 23, 2018When Queues Stop Working: Segment's Database-as-a-QueueSlack Engineering · Dec 6, 2017The Queue That Couldn't Drain: Kafka in Front of Redis at Slack