Pattern · seen in 3 breakdowns across 3 companies

Queue with Guaranteed Delivery

resilience

Definition

A queue with guaranteed delivery persists messages durably until they have been successfully consumed and acknowledged. The queue tolerates large backlogs, survives consumer failures, and does not drop messages under sustained pressure. The contract with producers is: once the queue acknowledges receipt, the message will eventually be delivered. The contract with consumers is: the message remains available (and potentially redelivered) until it's been processed and acknowledged.

The pattern's defining contrast is with the alternative often present in early architectures: an in-memory or lightly-persisted buffer that exhibits queue-like semantics under normal load but degrades to data loss under sustained pressure. Redis used as a queue is a common example — it works well as a buffer, but its eviction policies and memory constraints mean that under sufficient backpressure, messages disappear. Buffers like this are useful for many things, but they are not queues in the strict sense, because they do not guarantee delivery.

The principle the pattern enforces is simple but easy to violate: if your queue's failure mode under pressure is data loss, you don't have a queue, you have a buffer. Real queues persist. Whether through replicated disk-backed logs (Kafka, Kinesis), managed cloud services (PubSub, SQS, EventBridge), or carefully-operated message brokers (RabbitMQ with durable queues, Pulsar with persistent storage), the implementation must store messages durably enough to survive consumer failures, producer surges, and partial cluster outages.

The practical implication for system design is that any workflow where 'message lost' is a customer-visible problem requires a guaranteed-delivery queue between producers and consumers. Examples include indexing pipelines (lost messages mean unsearchable data), notification systems (lost messages mean failed user-visible alerts), event-sourced systems (lost messages mean state divergence), and any pipeline that performs work the system can't reconstruct from upstream state.

When it applies

01Indexing pipelines feeding search or analytics systems where lost messages produce visible gaps in downstream data
02Asynchronous workflows where the work cannot be reconstructed from upstream sources (the message is the only authoritative record of the work to be done)
03Notification, alerting, or webhook systems where lost messages mean failed user-visible outcomes that the system has already committed to delivering
04Inter-service event streams in event-driven architectures, especially when events trigger persistent state changes downstream
05Workloads where producer rate can occasionally exceed consumer capacity, and the correct behavior is for the queue to absorb the difference rather than for the system to lose data
06Any pipeline replacing an in-memory queue or unpersisted buffer that has exhibited message loss under load — this is one of the most common architectural upgrades distributed systems undergo

Tradeoffs

Guaranteed delivery typically costs latency. The queue must persist each message durably before acknowledging receipt to the producer, which means writes commit to disk (or to multiple replicas) before the producer can move on. For most workloads this latency is acceptable; for very latency-sensitive ones, the architectural choice may be different.
Backlog tolerance creates a different failure mode: instead of dropping messages, the queue accumulates them. A consumer outage can lead to a multi-hour backlog that the consumers then have to drain. The system must be designed for this — consumer scale-up under backlog, monitoring of queue depth, and graceful recovery from large backlogs are all required.
Storage costs grow with backlog size. A queue holding a 10-million-message backlog uses meaningful storage. Managed services charge for this; self-hosted systems consume disk for it. Long-running incidents that produce large backlogs can have surprising operational costs.
Exactly-once delivery semantics are usually a separate concern from guaranteed delivery. Most guaranteed-delivery queues provide at-least-once semantics; consumers must be idempotent (or the system must accept some duplicate processing). The combination of guaranteed delivery and exactly-once processing requires additional design work beyond the queue itself.
Operational complexity is real. Guaranteed-delivery queues require durable storage, replication, and careful operational care. The simplest version is a managed service, which trades cost for operational simplicity; self-hosted versions trade cost for control.

The same move, 3 ways

Every row is a production system that bet on this pattern — the note says how, in that system's own terms.

Discord
Discord Engineering
2025
The Redis-to-PubSub migration. Discord's original Redis indexing queue dropped messages under sustained pressure: a 'buffer', not a real 'queue'. PubSub guarantees delivery and holds large backlogs, so downstream failures now show up as slowdowns rather than lost data. The principle: if your queue's failure mode is losing data, you don't have a queue, you have a buffer. Real queues persist. Read the breakdown →
Meta
Engineering at Meta
2021
FOQS puts this whole property in one platform: items persist, leases turn consumer crashes into redeliveries, ack/nack give at-least-once semantics, and backlogs of hundreds of billions of items are cited as the system absorbing widespread downstream failure. Discord reached for the same guaranteed-delivery property in its PubSub service after Redis dropped messages under pressure; here Meta builds the property itself, as a shared multitenant platform. Read the breakdown →
Slack
Slack Engineering
2017
Delivery is guaranteed by never losing a job silently: the relay advances Kafka's position only after a job is safely in Redis, and re-enqueues a failed job instead of dropping it, so loss turns into delay. Two other articles here (Discord and Meta) make the same trade. The honest exception is at the front door, where the gateway acknowledges a write as soon as the lead Kafka broker has it, accepting a small, documented chance of loss in exchange for speed. Read the breakdown →

Problems this pattern answers

The walls where its breakdowns live — each opens the cross-company comparison.

Buffer degrades under backlogMETA · SLACK →Blast radius scales with cluster sizeDISCORD →