The Ledger Above the Log: Uber's Kafka Consumer Proxy
Uber runs one of the world's largest Kafka deployments (trillions of messages, petabytes daily), and more than 300 of its microservices use Kafka not for streaming but as a message queue, a job Kafka's core semantics quietly resist. Kafka allows only one consumer to work on a partition, and the safe pattern is to finish one message before starting the next, so each partition works like a single checkout lane: one worker, handling one message at a time, in order. When each job is a slow one-second call to a payment provider, reaching a thousand jobs per second means renting a thousand partitions, each nearly idle. And a single lane means one slow or stuck message holds up every healthy message behind it. Consumer Proxy is Uber's answer, built on top of Kafka rather than replacing it: a proxy that takes over talking to Kafka, hands each message to consumer services individually over gRPC, keeps its own record of which messages finished, and reports progress to Kafka the only way Kafka can record it: "all messages up to here are done."
Run a billing partition where Visa slows down and watch Mastercard's charges starve behind it — then turn on parallel dispatch and jam anyway, add the acknowledgment ledger and watch the watermark climb past stragglers, drop a poison pill to freeze the window, and nack it to the dead letter queue.
Problem
The post opens with the distinction its whole argument rests on. Kafka is stream-oriented: messages within a partition arrive in order, and that order is a guarantee. It is enforced by two rules: each partition is consumed by exactly one consumer in a group, and the recommended pattern is to fully process and commit one message before touching the next. (Autocommit, which lets Kafka mark messages done at fetch time before they are actually processed, stays off, because these workloads need at-least-once processing: every message handled at least once, even if that occasionally means twice.) Message queueing wants the opposite: unordered, point-to-point delivery where any worker can take any message. More than 300 microservices at Uber were running the message-queue workload on the streaming system's rules.
The post's example (labeled hypothetical, and kept that way here) is a billing queue: trip-completion events, a billing service charging payment providers, trip_1 to Visa and trip_2 to Mastercard in the same partition. The first pain, partition scalability, is arithmetic. A charge is a roughly one-second call to an external provider, and a partition that must finish each message before starting the next moves one message per second. Reaching 1,000 events per second therefore takes a 1,000-partition topic, while each partition could sustain about 10,000 messages per second (10 MB/s at 1 KB messages). That is 0.01% utilization, bought at scale: a cluster holds about 200,000 partitions in total, so thousand-partition topics cap out near 200 per cluster.
The second pain has a name: head-of-line blocking, where the first item in the lane delays everyone behind it. It comes in two flavors. Non-uniform latency: Visa's processing slows, and trip_2's Mastercard charge (entirely healthy) waits behind trip_1. Poison pills: a message that can never be processed (a permanent error, not a passing one) blocks its partition forever. Both built-in workarounds are rejected in the post. Autocommit converts blockage into data loss, because a restarted consumer skips past messages that were marked done but never processed, untenable for billing. And driving call latency down to a still-realistic 100 ms leaves a partition at ten messages per second.
Solution
Consumer Proxy sits between Kafka and every consumer, and its flow is a straight sequence: fetch messages from Kafka over its binary protocol, push each message individually to a consumer service's gRPC endpoint, receive a status code per message, track the results in its own ledger, and commit offsets back to Kafka when it is safe to. One thing does not change: each partition is still read by exactly one proxy node, so Kafka's view of the world is untouched. All the new freedom lives above it, in how the proxy fans messages out. Consumer services stop being Kafka clients entirely. No consumer groups, no partitions, just a gRPC handler.
The post is explicit that the proxy shape beat a client library on organizational grounds. One implementation serves Go, Java, Python, and NodeJS services alike, and upgrades ship once, in the proxy, instead of through months-long rollouts by 1,000+ service teams. Rebalances (Kafka pausing a consumer group to reassign partitions whenever members join or leave, which a rolling restart of a large service triggers over and over) touch a handful of proxy nodes instead of hundreds of instances. And a four-partition topic is no longer limited to four working instances (one per partition, the rest idle): the proxy spreads its messages across every instance of the service.
The mechanism ships in three moves, each killing one problem. Parallel processing within partitions: the proxy takes a batch and dispatches messages concurrently to any number of consumer instances, so partition count no longer caps consumer count. But naive parallelism still jams: the batch cannot be committed until its slowest member finishes, so the proxy cannot fetch more. The head-of-line block moves from the message to the batch. Out-of-order commit is the load-bearing invention, and it rests on one distinction: a Kafka commit at offset N declares everything up to N done, while a proxy acknowledgment marks exactly one message done. Consumers ack individual messages to the proxy, and the proxy commits to Kafka only when an unbroken run of messages from the last committed offset is fully acked. That committed offset acts like a watermark, a line marking how far everything is finished, and it climbs past stragglers without waiting for them. The post is honest about the leftover cost: a message can be acked but not yet committed, and after a rebalance Kafka delivers those messages again. Uber accepts the duplicates because its consumer services already detect and drop them. That is at-least-once delivery, resting on a company-wide habit of idempotency: handling the same message twice has the same effect as handling it once. One blockage survives: a poison pill can never be acked, so the watermark stops beneath it and, once the tracker fills, fetching stops too.
The dead letter queue adds the missing word. A consumer flags an unprocessable message with a gRPC error code; the proxy persists it to a separate DLQ Kafka topic and records it as negatively acknowledged, and the watermark treats nacked messages like acked ones and moves past them. The stuck message no longer blocks its partition; it becomes a task for an operator later, with tooling to replay (merge) or delete (purge) it. Around the push model sits flow control, because pushing flips backpressure, the "slow down, I'm full" signal: a pull-based consumer cannot be overwhelmed, since it takes work only when ready, while a push proxy sets the pace and must be told how fast is safe. The tracker's size (how many messages may be in flight and unacked at once) is exactly that pacing limit, set per service at onboarding along with processing timeouts. Returned gRPC codes adjust push speed, and a circuit breaker stops pushing entirely when a service is down, so an outage doesn't flush healthy messages into the DLQ.
The alternatives survey explains why this was built, not bought, in 2018–2019. Confluent's REST proxy solved neither pain and had no DLQ. Kafka Connect could have hosted the features, but its rebalancing then stopped the world on every membership change (incremental rebalancing arrived later, via the Kafka improvement proposal KIP-415), which fought Uber's sub-10-second end-to-end latency targets; and DLQ replay needs bounded consumption (read a topic up to a fixed offset, then stop), which Connect lacked. By publication, Consumer Proxy was the primary async-queueing path at Uber, with hundreds of services onboarded; it was later open-sourced as uForwarder.
Tradeoffs
- Kafka's partition bundles three things into one unit: message order, how many workers can run, and how progress is recorded. A queueing workload needs none of that bundling and pays for all of it. The bill is the arithmetic: more throughput under the native pattern means more partitions, a thousand of them each moving one message per second against a capacity of roughly 10,000. And because a cluster can hold only about 200,000 partitions in total, there is room for only about 200 such topics. Using a stream as a queue means renting rules you must then engineer around; the whole article is the engineering-around.
- Splitting acknowledge from commit means the truth about progress lives in two places. The proxy knows exactly which messages finished; Kafka knows only the done-up-to-here line. In between sit messages that finished but are not yet covered by that line, and after a rebalance Kafka delivers them again. Uber accepts the duplicates because its consumer services already remove them, but that is the fine print: the proxy's at-least-once promise depends on every consumer team keeping its deduplication working, forever.
- The DLQ converts blocking into deferral, not resolution. A stuck message stops holding up everything behind it, but it is not fixed; it waits in the DLQ for a person to replay it or delete it. And the nack path is explicit: a consumer must return the error code that sends a message to the DLQ. A handler that hangs instead of failing says nothing, so timeouts are still needed to turn silence into a decision.
- Push flips who controls the pace. A pull-based consumer can never be overwhelmed, because it asks for the next message only when it is ready. A push proxy sends, so it must be told how fast is safe: a per-service cap on in-flight messages, timeouts, slow-down signals read from gRPC codes, and a circuit breaker that stops sending when a service is down. The post's own Next Steps admits this tuning is hard enough that self-adjusting flow control, like TCP's, is the roadmap. Freedom from the partition was bought by taking over the pacing job the partition used to do badly.
- A proxy beats a client library organizationally and charges an infrastructure toll. One codebase serves every language; the Kafka team upgrades it in one place instead of waiting on a thousand service teams; a rebalance touches a few proxy nodes instead of hundreds of instances. In exchange, every message now passes through one more service on its way to the consumer, and that service's latency and availability are added to every consumer's end-to-end delay (the same sub-10-second targets that disqualified Kafka Connect).
- Building over adopting was a dated decision, honestly dated. In the 2018–2019 landscape, Confluent's REST proxy lacked the semantics, and Kafka Connect lacked incremental rebalancing (pre-KIP-415) and the bounded consumption DLQ replay needs, so Uber built. The ecosystem kept moving (KIP-429 and KIP-415 appear in the post's own citations), and the eventual open-sourcing as uForwarder reads as the decision's second act: what had to be proprietary in 2019 became shared infrastructure once proven.
Patterns in this article
- Selective Acknowledgment
Minted from the post's central invention, under the name the networking world already uses: per-item acknowledgments tracked in a ledger above a substrate whose only durable notion of progress is a contiguous watermark — TCP SACK's exact shape, applied to a commit log. The proxy distinguishes acknowledge (this one message) from commit (everything below this offset), advances Kafka's offset only through contiguous acked-or-nacked ranges, and accepts the residue: rebalance re-fetches the acked-but-uncommitted, deduplicated downstream.
- Dead Letter Queue
The companion that completes selective acknowledgment's vocabulary: an unprocessable message is negatively acknowledged, persisted to a DLQ topic, and the watermark passes over it — converting an indefinite partition blockage into a deferred operator decision with merge/purge tooling. The circuit breaker guards the pattern's failure mode: when the consumer is entirely down, stop pushing, so an outage doesn't launder healthy messages into the dead letter queue.
- Fault Isolation
The proxy-versus-library section is an isolation argument in organizational clothing: decoupling the few message-consuming nodes from the hundreds of processing instances contains consumer-group rebalance storms to a small blast radius, and puts upgrade authority in one team's hands instead of a thousand services'. The isolation boundary here is drawn around operational churn, not just failure.
Also solving this
Other systems in behindscale's Buffer degrades under backlog class: