From Static Rate Limiting to Intelligent Load Management
Uber's load management evolved through three phases inside its distributed databases (Docstore and Schemaless). Quota-based rate limiting failed because it lived at the stateless query layer. CoDel-based queue management stabilized the storage engine but treated all traffic equally. Cinnamon combines priority-aware shedding (t0-t5 tiers), PID-controlled dynamic thresholds, and a pluggable signal architecture (BYOS) — letting any overload signal feed into a single, coherent control loop. The result is +80% throughput under overload, 70% lower P99 latency, and a 93% reduction in goroutine count during overload events.
Break one storage node four ways, then walk each failure through all three generations of overload control — and watch who each one drops.
Problem
Uber's distributed databases — Docstore and Schemaless, built on MySQL — span thousands of clusters, store tens of petabytes, and serve tens of millions of requests per second, with traffic ranging from latency-critical user-facing queries (rides, payments) to low-priority pipelines, aggregators, and batch jobs. At this scale overloads don't stay local: a brief spike ripples outward as downstream services time out and retries pile up, and in a multitenant system a single noisy tenant can degrade everyone. The system needs to shed load gracefully without compromising critical traffic or amplifying the problem through retry storms.
The first attempt put rate limiting at the stateless query engine layer, with Redis tracking per-tenant usage against capacity-unit quotas. This failed in multiple ways: the Redis call added latency and a new point of failure, the cost model couldn't distinguish a full table scan from a single-row read, and the stateless layer had no visibility into the actual health of thousands of storage partitions behind it. Static quotas required constant manual tuning that always lagged real traffic patterns.
The second iteration moved overload management into the storage engine itself, using CoDel for queue-based admission (shedding by request wait time, switching to LIFO under pressure), Scorecard for per-tenant concurrency limits, and a set of Regulators for write bytes, hot keys, memory pressure, and goroutine count. This stabilized the system but introduced a different problem: it was priority-blind. A critical ride pricing query was shed with the same probability as a batch analytics job. Mass rejection under load also caused thundering herd retry storms.
Solution
Cinnamon is the unified load manager built on three principles: priority awareness, dynamic adaptation, and pluggable signals.
Priority awareness comes from a t0-t5 tier system applied to every request. Critical infrastructure (t0) is shed last (essentially never). User-facing traffic like ride pricing and payments (t1) is protected aggressively. Important-but-deferrable work (t2) is shed when the system is stressed. Async jobs, pipelines, and aggregators (t3-t5) are first to go. When Cinnamon needs to reduce throughput, it walks up the tier ladder from lowest to highest, only touching higher tiers if pressure persists.
Dynamic adaptation comes from a PID controller. Where CoDel relied on fixed queue timeouts and static inflight concurrency limits — aggressively rejecting all requests once a fixed wait time, like 5 milliseconds, was exceeded — Cinnamon adapts its queue timeout thresholds from P90 latency metrics and continuously adjusts inflight limits with its Auto Tuner, reacting to realtime latency and error-rate signals. The PID-based control (detailed in Uber's companion posts on Cinnamon and its PID controller) lets the system absorb pressure without overreacting: admission adjusts smoothly instead of snapping between open and closed, preventing the premature-shedding-then-retry cycle that fed thundering herds under the static design. The post's own image: without PID regulation, shedding acts like a hammer; with it, a dimmer switch.
Pluggable signals (Bring Your Own Signal — BYOS) is the architectural multiplier. Each overload signal — inflight concurrency, follower commit lag, write bytes volume, hot partition key, memory pressure, goroutine count — used to be its own independent rate limiter. They could split-brain: a commit-lag limiter throttling writes while a concurrency controller tried to admit them. BYOS makes Cinnamon a platform: every signal feeds the same unified decision loop, the controller sees the full picture, and the resulting decision is coherent across all signals.
The production results published in the article: roughly +80% throughput under overload (QPS averaging 5,400 versus 3,000), P99 latency dropping from ~3.1s to ~1.0s, and resource consumption dropping dramatically (goroutines from 150K to 10K, heap from 5-6GB to 1GB). The improvements come primarily from the qualitative shift — Cinnamon doesn't just shed traffic, it sheds the right traffic at the right rate.
Tradeoffs
- PID regulation concentrates the tuning problem rather than eliminating it. Per-service static configuration disappears — Cinnamon's stated goal is no per-service tuning at all — but the controller itself had to be made stable at the platform level, and Uber's companion PID post notes that earlier iterations of the controller didn't achieve a consistent rejection span across low- and high-traffic services. The calibration cost is paid once, centrally, by the platform team instead of continuously by every service team; organizations adopting the pattern should budget for that platform-level investment.
- Priority tiers require system-wide classification discipline. Every request in every service must be tagged with a priority tier, and this taxonomy must be maintained as new use cases emerge. Misclassification (tagging analytics queries as user-facing) defeats the protection; honest classification is an ongoing editorial responsibility for the whole engineering organization.
- BYOS adds complexity at the integration layer. Each new signal must be normalized into a form the unified controller can consume, with units and scales that compose correctly. A signal that produces values orders of magnitude different from concurrency counts can destabilize the control loop. The platform's flexibility comes with required engineering rigor at the seams.
- The whole system depends on accurate per-request priority metadata. If priority tagging is wrong (or missing), Cinnamon's intelligence degenerates to roughly CoDel-equivalent behavior. The system is robust to overload but not robust to classification failure.
Patterns in this article
- Priority-Aware Load Shedding
Cinnamon's t0-t5 tier system is the canonical implementation: every request carries a priority tag, and shedding decisions walk up the tier ladder from lowest to highest. Critical user-facing traffic (ride pricing, payments) is protected while async analytics absorbs the cost of overload. The pattern's value depends on accurate priority classification system-wide — an editorial discipline, not just an engineering one.
- Feedback-Controlled Load Management
Cinnamon is the worked instance: a PID loop over queue and latency signals smoothly adjusts admission instead of snapping a static threshold, and the BYOS design extends the same loop to any normalized overload signal — commit lag, write bytes, memory pressure — unifying what were previously competing per-signal limiters into one coherent decision path.
Also solving this
Other systems in behindscale's Priority-blind load shedding class: