Selective Acknowledgment

throughput

Definition

When a substrate's only durable notion of progress is a contiguous watermark — a commit offset, a sequence number — but work items complete out of order, track per-item acknowledgments in a ledger above the substrate and advance the durable watermark only through contiguous acknowledged ranges. Items are dispatched and completed in any order; stragglers stop blocking everything behind them; the substrate below never learns that order was violated, because it only ever sees the watermark move forward. TCP's SACK option is the pattern's namesake and oldest instance; a Kafka consumer proxy distinguishing per-message 'acknowledge' from offset 'commit' is the same shape applied to a log.

The pattern's honest residue is the gap between the two ledgers: items acknowledged above but not yet committed below are replayed if the upper ledger is lost (a crash, a rebalance), so the pattern delivers at-least-once and delegates exactly-once to downstream idempotency. And a permanently unacknowledgeable item still jams the watermark beneath it — which is why this pattern travels with a negative-acknowledgment route (a dead letter queue) that lets the watermark pass over declared failures. Boundary against Database-as-a-Queue: that pattern replaces the ordered substrate with one that supports per-item state natively; this one keeps the substrate and layers the per-item state above it — cheaper to adopt, permanently two-ledgered.

When it applies

  • Ordered logs or streams (Kafka, event logs, sequenced replication) used for workloads where item order is meaningless and per-item completion varies widely
  • Systems where replacing the substrate is impractical but head-of-line blocking is intolerable
  • Windowed protocols of any kind where retransmitting everything past a single loss wastes the pipe — the TCP SACK case generalized

Tradeoffs

  • Two ledgers means a reconciliation gap: acknowledged-but-uncommitted items replay on upper-ledger loss, so downstream idempotency becomes a hard dependency, not a nicety
  • The tracking window is finite; a permanently stuck item exhausts it and stalls all progress unless a negative-acknowledgment escape exists
  • The upper ledger is new stateful infrastructure on the hot path — its size, timeout, and recovery behavior are now tuning surface the plain consumer never had

Seen in

  • Uber EngineeringAug 31, 2021

    The Ledger Above the Log: Uber's Kafka Consumer Proxy

    Minted from the post's central invention, under the name the networking world already uses: per-item acknowledgments tracked in a ledger above a substrate whose only durable notion of progress is a contiguous watermark — TCP SACK's exact shape, applied to a commit log. The proxy distinguishes acknowledge (this one message) from commit (everything below this offset), advances Kafka's offset only through contiguous acked-or-nacked ranges, and accepts the residue: rebalance re-fetches the acked-but-uncommitted, deduplicated downstream.