Queue with Guaranteed Delivery

resilience

Definition

A queue with guaranteed delivery persists messages durably until they have been successfully consumed and acknowledged. The queue tolerates large backlogs, survives consumer failures, and does not drop messages under sustained pressure. The contract with producers is: once the queue acknowledges receipt, the message will eventually be delivered. The contract with consumers is: the message remains available (and potentially redelivered) until it's been processed and acknowledged.

The pattern's defining contrast is with the alternative often present in early architectures: an in-memory or lightly-persisted buffer that exhibits queue-like semantics under normal load but degrades to data loss under sustained pressure. Redis used as a queue is a common example — it works well as a buffer, but its eviction policies and memory constraints mean that under sufficient backpressure, messages disappear. Buffers like this are useful for many things, but they are not queues in the strict sense, because they do not guarantee delivery.

The principle the pattern enforces is simple but easy to violate: if your queue's failure mode under pressure is data loss, you don't have a queue, you have a buffer. Real queues persist. Whether through replicated disk-backed logs (Kafka, Kinesis), managed cloud services (PubSub, SQS, EventBridge), or carefully-operated message brokers (RabbitMQ with durable queues, Pulsar with persistent storage), the implementation must store messages durably enough to survive consumer failures, producer surges, and partial cluster outages.

The practical implication for system design is that any workflow where 'message lost' is a customer-visible problem requires a guaranteed-delivery queue between producers and consumers. Examples include indexing pipelines (lost messages mean unsearchable data), notification systems (lost messages mean failed user-visible alerts), event-sourced systems (lost messages mean state divergence), and any pipeline that performs work the system can't reconstruct from upstream state.

When it applies

  • Indexing pipelines feeding search or analytics systems where lost messages produce visible gaps in downstream data
  • Asynchronous workflows where the work cannot be reconstructed from upstream sources (the message is the only authoritative record of the work to be done)
  • Notification, alerting, or webhook systems where lost messages mean failed user-visible outcomes that the system has already committed to delivering
  • Inter-service event streams in event-driven architectures, especially when events trigger persistent state changes downstream
  • Workloads where producer rate can occasionally exceed consumer capacity, and the correct behavior is for the queue to absorb the difference rather than for the system to lose data
  • Any pipeline replacing an in-memory queue or unpersisted buffer that has exhibited message loss under load — this is one of the most common architectural upgrades distributed systems undergo

Tradeoffs

  • Guaranteed delivery typically costs latency. The queue must persist each message durably before acknowledging receipt to the producer, which means writes commit to disk (or to multiple replicas) before the producer can move on. For most workloads this latency is acceptable; for very latency-sensitive ones, the architectural choice may be different.
  • Backlog tolerance creates a different failure mode: instead of dropping messages, the queue accumulates them. A consumer outage can lead to a multi-hour backlog that the consumers then have to drain. The system must be designed for this — consumer scale-up under backlog, monitoring of queue depth, and graceful recovery from large backlogs are all required.
  • Storage costs grow with backlog size. A queue holding a 10-million-message backlog uses meaningful storage. Managed services charge for this; self-hosted systems consume disk for it. Long-running incidents that produce large backlogs can have surprising operational costs.
  • Exactly-once delivery semantics are usually a separate concern from guaranteed delivery. Most guaranteed-delivery queues provide at-least-once semantics; consumers must be idempotent (or the system must accept some duplicate processing). The combination of guaranteed delivery and exactly-once processing requires additional design work beyond the queue itself.
  • Operational complexity is real. Guaranteed-delivery queues require durable storage, replication, and careful operational care. The simplest version is a managed service, which trades cost for operational simplicity; self-hosted versions trade cost for control.

Seen in

  • Discord EngineeringApr 24, 2025

    How Discord Indexes Trillions of Messages

    The Redis-to-PubSub migration. Discord's original Redis indexing queue dropped messages under sustained pressure — a 'buffer' rather than a 'queue.' PubSub provides guaranteed delivery and tolerates large backlogs, which means downstream failures now manifest as slowdowns rather than data loss. The pattern's principle: if your queue's failure mode is data loss, you don't have a queue, you have a buffer. Real queues persist.

  • Engineering at MetaFeb 22, 2021

    Delay, Not Loss: FOQS, Meta's Trillion-Item Priority Queue

    FOQS is the pattern as a product: persistent items, leases that convert consumer crashes into redeliveries, ack/nack with at-least-once semantics, and backlogs of hundreds of billions of items cited as the system absorbing widespread downstream failure. Second company: Discord reached for this property in PubSub after Redis dropped messages under pressure; Meta built the property itself, as a multitenant platform.