The Ledger Above the Log: Uber's Kafka Consumer Proxy
Uber runs one of the world's largest Kafka deployments — trillions of messages and petabytes daily — and more than 300 microservices use it not for streaming but as a pub-sub message queue, a job Kafka's core semantics quietly resist. After five years scaling that use from 1 to 12 million messages per second, two pains had names: partition scalability (a one-second RPC per message means a thousand partitions to reach a thousand events per second, each running at one message per second against the ~10,000 it can sustain) and head-of-line blocking (per-partition ordering means one slow or poisoned message stalls everything behind it). Consumer Proxy is the answer built above the log rather than instead of it: a push proxy that fetches from Kafka, dispatches each message individually to consumer gRPC endpoints, tracks per-message acknowledgments and negative acknowledgments in its own ledger, parks poison pills in a dead letter queue topic, and commits to Kafka only the contiguous watermark — so the partition stops being the unit of parallelism, order, and progress all at once.
Run a billing partition where Visa slows down and watch Mastercard's charges starve behind it — then turn on parallel dispatch and jam anyway, add the acknowledgment ledger and watch the watermark climb past stragglers, drop a poison pill to freeze the window, and nack it to the dead letter queue.
Problem
The post opens by drawing the distinction its whole argument rests on. Kafka is stream-oriented: message ordering is a fundamental semantic property, enforced by exclusive partition-to-consumer assignment and the recommended consumer pattern — read one message, process it, and do not process the next until the previous one is committed (autocommit stays off, because at-least-once processing is semantically required for these workloads). Message queueing wants the opposite: unordered, point-to-point, any-consumer delivery where no pair of messages has a meaningful order. More than 300 microservices at Uber were running the second workload on the first system's rules.
The hypothetical the post uses — labeled hypothetical, and kept that way here — is a billing queue: trip-completion events, a billing service charging third-party providers, trip_1 to Visa and trip_2 to Mastercard in the same partition. Partition scalability is arithmetic: at a one-second RPC per charge, reaching 1,000 events per second under the native pattern requires a 1,000-partition topic, each partition delivering one event per second against the roughly 10,000 messages per second (10 MB/s at 1 KB messages) it could sustain — while the cluster's ~200,000-partition budget caps such topics at about 200 per cluster. The resource is rented at 0.01% utilization because parallelism is chained to partition count.
Head-of-line blocking is the same coupling felt at runtime, in two flavors. Non-uniform latency: Visa's processing latency spikes, and trip_2's Mastercard charge — entirely healthy — waits behind trip_1. Poison pills: a message that cannot be processed due to non-transient errors blocks its partition indefinitely. Kafka's native escapes are both rejected on the record: autocommit converts blockage into data loss (a restarted consumer resumes past auto-committed, never-processed messages — untenable for billing), and driving RPC latency down to a still-reasonable 100ms leaves a partition at ten messages per second.
Solution
Consumer Proxy is a push proxy interposed between Kafka and every consumer: it fetches messages over the Kafka binary protocol, pushes each message separately to a consumer service's gRPC endpoint, receives a gRPC status code per message, aggregates results, and commits offsets to Kafka when it is safe to. Consumers stop being Kafka clients at all — no consumer groups to manage, no partitions to think about, just a gRPC handler. The post is explicit that the proxy shape beat the client-library shape on organizational physics: one implementation serves a polyglot fleet of Go, Java, Python, and NodeJS services; upgrades stop requiring months-long campaigns across 1,000+ microservices; decoupling the few consuming nodes from the hundreds of processing instances contains consumer-group rebalance storms (with the proxy running its own rebalance logic); and a topic with four partitions can now spread work almost evenly across a service's every instance instead of hot-spotting four of them.
The mechanism ships in three moves, each named for the problem it kills. Parallel processing within partitions: the proxy consumes a batch and dispatches messages concurrently to any number of consumer instances — partition count no longer bounds consumer count. But naive parallelism still jams: the batch cannot be committed until its first message completes, so the proxy cannot fetch more. Out-of-order commit is the load-bearing invention: consumer services acknowledge single messages to the proxy — where a Kafka commit asserts everything at lower offsets is done, a proxy acknowledgment marks exactly one message — and the proxy's tracker commits to Kafka only when a contiguous range from the last committed offset is fully acknowledged. Progress continues while stragglers finish. The post prices the design's residue honestly: acknowledged-but-uncommitted messages are re-fetched after a rebalance, and Uber accepts the duplication because consumer services carry built-in deduplication regardless — at-least-once delivery leaning on the ecosystem's idempotency discipline. And one blockage survives: a poison pill can never be acknowledged, so the tracker's watermark jams beneath it and fetching eventually stops.
The dead letter queue completes the ledger's vocabulary. A consumer signals an unprocessable message with a gRPC error code; the proxy persists it to a DLQ Kafka topic and records it as negatively acknowledged — and the watermark advances over acked-or-nacked messages alike. Poison pills stop costing the partition and start costing an operator later, with tooling to merge (replay) or purge the DLQ. Around the push model sits flow control, because pushing inverts backpressure ownership: per-service tracker sizes and processing timeouts set at onboarding, push speed adjusted from returned gRPC codes, and a circuit breaker that stops pushing entirely when a consumer service is down — so an outage doesn't launder healthy messages into the DLQ. The post's survey of alternatives explains why this was built, not bought, in the 2018–2019 landscape: Confluent's REST proxy solved neither partition scalability nor single-message acknowledgment and offered no DLQ; Kafka Connect could have hosted the features but pre-KIP-415 rebalancing fought their sub-10-second P99 end-to-end targets, and DLQ redelivery needed bounded consumption Connect lacked. By publication, Consumer Proxy was the primary async-queueing path at Uber with hundreds of services onboarded — later open-sourced as uForwarder, with over 1,000 consumer services on it.
Tradeoffs
- Kafka's partition couples ordering, parallelism, and progress into one unit, and queueing pays for all three while needing none. The 1,000-partitions-at-1-message-per-second arithmetic is the coupling as a bill: to buy consumer throughput under the native pattern you buy partitions, and the cluster's partition budget (~200K) turns underutilization into a hard ceiling of ~200 such topics. Using a stream as a queue means renting semantics you must then engineer around — the whole article is the engineering-around.
- Acknowledge-versus-commit splits truth into two ledgers, and the split has a residue. The proxy's tracker holds per-message truth; Kafka holds only the contiguous watermark; between them live acknowledged-but-uncommitted messages that redeliver after a rebalance. Uber's acceptance is candid and load-bearing: duplication is fine because consumer services dedupe anyway — the proxy's at-least-once contract quietly rests on the organization-wide idempotency discipline that is its own class of problem.
- The DLQ converts blocking into deferral, not resolution. A poison pill stops costing every message behind it and starts costing an operator later — merge or purge is a human decision queue with tooling, not an automated outcome. And the nack path is explicit: a consumer must return the gRPC error code that routes a message to the DLQ, so a handler that hangs rather than fails still needs the timeout machinery to convert silence into a decision.
- Push inverts backpressure ownership, and the proxy inherits congestion control. Pull-based consumers self-pace by construction; a push proxy must be told — per-service tracker sizes and timeouts at onboarding, push-speed adjustment from gRPC codes, a circuit breaker for downed consumers. The post's Next Steps concedes the tuning is hard enough that adaptive, TCP-congestion-style flow control is the roadmap: partition-freedom was bought by taking on the pacing problem the partition used to solve badly.
- A proxy beats a client library on organizational physics and charges an infrastructural toll. One implementation for a polyglot fleet, upgrades without thousand-service campaigns, rebalance blast radius contained to a few consuming nodes — and every message now crosses an additional stateful hop, owned by the Kafka team, whose latency and availability sit inside every consumer's end-to-end P99 (the same sub-10-second targets that disqualified Kafka Connect).
- Building over adopting was a dated decision, honestly dated. In the 2018–2019 landscape, the REST proxy lacked the semantics and Connect lacked incremental rebalancing (pre-KIP-415) and bounded consumption for DLQ redelivery — so Uber built. The ecosystem kept moving (KIP-429, KIP-415 land in the post's own citations), and the eventual open-sourcing as uForwarder reads as the decision's second act: what had to be proprietary in 2019 becomes shared infrastructure once proven.
Patterns in this article
- Selective Acknowledgment
Minted from the post's central invention, under the name the networking world already uses: per-item acknowledgments tracked in a ledger above a substrate whose only durable notion of progress is a contiguous watermark — TCP SACK's exact shape, applied to a commit log. The proxy distinguishes acknowledge (this one message) from commit (everything below this offset), advances Kafka's offset only through contiguous acked-or-nacked ranges, and accepts the residue: rebalance re-fetches the acked-but-uncommitted, deduplicated downstream.
- Dead Letter Queue
The companion that completes selective acknowledgment's vocabulary: an unprocessable message is negatively acknowledged, persisted to a DLQ topic, and the watermark passes over it — converting an indefinite partition blockage into a deferred operator decision with merge/purge tooling. The circuit breaker guards the pattern's failure mode: when the consumer is entirely down, stop pushing, so an outage doesn't launder healthy messages into the dead letter queue.
- Fault Isolation
The proxy-versus-library section is an isolation argument in organizational clothing: decoupling the few message-consuming nodes from the hundreds of processing instances contains consumer-group rebalance storms to a small blast radius, and puts upgrade authority in one team's hands instead of a thousand services'. The isolation boundary here is drawn around operational churn, not just failure.
Also solving this
Other systems in behindscale's Buffer degrades under backlog class: