Problem · seen at 5 companies

Buffer degrades under backlog

The queue you added to protect your system becomes the thing that takes it down.

The wall

A buffer that stops absorbing exactly when the backlog it exists for arrives — whether the buffer's substrate degrades under the backlog, or its access semantics let the backlog capture the buffer. 5 teams hit this wall; the breakdowns below are the evidence.

Same wall, 5 systems

Uber
Uber Engineering
Kafka's per-partition ordering lets one stuck message stall every message behind it. Read the breakdown →
Meta
Engineering at Meta
The queue's own substrate slows down exactly as the backlog it must absorb grows. Read the breakdown →
DoorDash
DoorDash Engineering Blog
Under peak bursts RabbitMQ throttled its publishers, and the slowdown cascaded upstream. Read the breakdown →
Segment
Segment Blog
Hundreds of destinations fail daily; a shared queue makes every failure everyone's backlog. Read the breakdown →
Slack
Slack Engineering
A full Redis queue couldn't dequeue, because draining required the very memory the backlog had consumed. Read the breakdown →

Patterns in this class

Every breakdown