From Static Rate Limiting to Intelligent Load Management
Uber's load management for its distributed databases (Docstore and Schemaless) evolved through three phases. Quota-based rate limiting failed because it ran too far from the storage nodes, up in the stateless query layer where overload doesn't actually show up. CoDel, a queue-trimming algorithm borrowed from networking that sheds requests by how long they have waited, stabilized the storage engine but treated a ride request and a batch job as equals. Cinnamon, the current generation, adds the judgment: it sheds by priority tier (t0, most critical, down to t5), continuously nudges its rejection threshold up and down against measured overload using a PID controller (a self-correcting loop, explained below), and accepts any overload symptom as a plug-in signal through a bring-your-own-signal design (BYOS). The result under overload: about 80% more throughput, roughly 70% lower P99 latency, and far fewer goroutines (Go's lightweight threads) piling up during overload events.
Break one storage node four ways, then walk each failure through all three generations of overload control — and watch who each one drops.
Problem
Uber's distributed databases, Docstore and Schemaless, are built on MySQL. They span thousands of clusters, store tens of petabytes, and serve tens of millions of requests per second, with traffic ranging from latency-critical user-facing queries (rides, payments) to low-priority pipelines, aggregators, and batch jobs. At this scale overloads don't stay local: a brief spike ripples outward as downstream services time out and retries pile up. The system needs to shed load gracefully, without dropping critical traffic or amplifying the problem through retry storms.
The first attempt put rate limiting at the stateless query engine layer, with a central Redis tracking each tenant's usage against a capacity-unit budget. It failed in several ways. The Redis call added latency and a new point of failure. The cost model couldn't tell a full table scan from a single-row read, so a heavy query and a light one were charged the same. And the stateless layer had no visibility into the actual health of the thousands of storage partitions behind it, so it was deciding what to shed without knowing what was actually overloaded. Static quotas also needed constant manual tuning that always lagged real traffic. The lesson the team drew: overload control has to live as close to the storage nodes as possible.
The second attempt moved control into the storage engine itself, using CoDel for queue-based admission (shedding by how long a request has waited, and switching to last-in-first-out under pressure so fresh requests aren't stuck behind stale ones), Scorecard for per-tenant concurrency caps, and a set of regulators for write bytes, hot keys, memory pressure, and goroutine count. This stabilized the system but introduced a different problem: it was priority-blind. A critical ride-pricing query was shed with the same probability as a batch analytics job. And because rejection was abrupt, mass rejections under load triggered thundering-herd retry storms, where everything that was dropped retries at once and re-overloads the system.
Solution
Cinnamon is the unified load manager, built on three principles: priority awareness, dynamic adaptation, and pluggable signals.
Priority awareness comes from a t0-t5 tier applied to every request. Critical infrastructure (t0) is shed last, essentially never. User-facing traffic like ride pricing and payments (t1) is protected aggressively. Important-but-deferrable work (t2) is shed when the system is stressed. Async jobs, pipelines, and aggregators (t3 to t5) go first. When Cinnamon needs to reduce load, it walks up the tier ladder from lowest to highest, only touching higher tiers if the pressure persists.
Dynamic adaptation comes from a PID controller. A PID controller is a small piece of control math that steers a system toward a target by combining three reactions to the current error: how far off it is right now (the P, proportional), how long it has been off (the I, integral), and which direction it is heading (the D, derivative). Adding those together gives a correction that is firm when the gap is large, patient when a small gap lingers, and gentle when things are already improving. A home thermostat and a car's cruise control work the same way.
Cinnamon points that machinery at admission. Where CoDel used a fixed queue timeout and a static concurrency limit, rejecting everything the moment a set wait time like 5 milliseconds was exceeded, Cinnamon adapts its timeout from live P90 latency and continuously adjusts how many requests it allows in flight, reacting to real-time latency and error signals. Because the controller nudges rather than switches, admission eases open and closed instead of snapping fully on or fully off, which prevents the premature-shedding-then-retry cycle that fed the thundering herds. The post's own image: without the controller, shedding acts like a hammer; with it, a dimmer switch.
Pluggable signals, which Uber calls Bring Your Own Signal (BYOS), is the architectural multiplier. Overload shows up as many different symptoms: inflight concurrency, write-byte volume, hot partition keys, memory pressure, goroutine count, and follower commit lag, which is a leader having to wait for its backup copies to catch up. Each of these used to be its own independent limiter, and they could contradict each other: one throttling writes while another tried to admit them, a split-brain where two controllers make opposite decisions about the same traffic at the same moment. BYOS makes Cinnamon a platform instead. Every signal feeds the same decision loop, so the controller sees the whole picture at once and the shedding decision stays coherent no matter which symptom raised the alarm.
The production results in the article: roughly 80% more throughput under overload (QPS averaging 5,400 versus 3,000), P99 latency dropping from about 3.1s to 1.0s, and resource use dropping sharply (goroutines from 150K to 10K, heap from 5-6GB to 1GB). The gains come mostly from the qualitative shift: Cinnamon doesn't just shed traffic, it sheds the right traffic at the right rate.
Tradeoffs
- PID control doesn't remove the tuning work, it moves it. Under the old system, every service team had to hand-tune its own static limits and re-tune them as traffic changed, forever. Cinnamon's goal is to end that: no per-service tuning at all. But the controller that replaces it has to be tuned once, carefully, for the whole platform, and getting that right was hard. Uber's companion post notes that early versions shed too aggressively on quiet services and not aggressively enough on busy ones, until the controller was calibrated to behave consistently across both. So the cost doesn't vanish; it shifts from many teams tuning constantly to one platform team tuning once. Anyone adopting this pattern should plan for that one-time, central investment.
- Priority tiers require system-wide classification discipline. Every request in every service has to be tagged with a tier, and that taxonomy has to be maintained as new use cases appear. Mis-tagging (marking analytics as user-facing) defeats the protection, so honest classification becomes an ongoing editorial responsibility for the whole engineering organization, not just an engineering one.
- BYOS adds complexity at the seams. Each new signal has to be normalized into a form the one controller can consume, on units and scales that compose correctly. A signal that produces values orders of magnitude larger than concurrency counts can destabilize the whole loop. The platform's flexibility comes with required rigor at every integration point.
- The whole system leans on accurate per-request priority metadata. If the tagging is wrong or missing, Cinnamon's intelligence degrades to roughly CoDel-level behavior: it will still keep the system up, but it will protect the wrong things. The design is robust to overload but not robust to classification failure.
- Failing fast means throwing away work you have already partly paid for. The switch to last-in-first-out under pressure, and rejecting early rather than holding requests until they expire, is the right call for responsiveness (it avoids memory bloat and wasted effort on stale requests), but it does mean a request that waited in the queue and then gets dropped consumed capacity for nothing. Fast rejection trades a little wasted work now to avoid a lot of wasted work later, and that trade is only worth it because holding requests to expiry is worse.
- Placing control at the storage layer ties overload protection to the storage engine. The lesson that the best shedding decisions happen where the state lives is the same reason the load manager can't be a reusable layer sitting above many systems: it has to live inside the engine that has the context. That is the exact mirror of why the first quota design failed for being too far from the storage nodes, and it means each stateful system that wants this protection has to grow its own copy rather than share one.
Patterns in this article
- Priority-Aware Load Shedding
Every request carries a priority tier, and when the system has to drop load it starts from the lowest tier and works up, so batch and analytics traffic absorbs the cost while rides and payments are protected. Cinnamon's t0-t5 tiers are a full worked example. The catch is that this only works if the tiers are honest: the pattern's whole value rests on every team classifying its traffic correctly, which is an editorial discipline across the organization, not just an engineering one.
- Feedback-Controlled Load Management
Instead of a fixed threshold that snaps fully open or fully shut, a controller adjusts how much to admit smoothly and continuously, using live latency and error signals as feedback (the same idea as a thermostat holding a temperature). Cinnamon's PID controller is the worked example, and its BYOS design extends the same loop to any overload signal (commit lag, write bytes, memory pressure), so what used to be several competing limiters becomes one coherent decision.
Also solving this
Other systems in behindscale's Priority-blind load shedding class: