Feedback-Controlled Load Management

throughput

Definition

Static limits are perpetually wrong. A fixed concurrency cap, queue timeout, or shed threshold encodes one moment's understanding of capacity into a number that traffic immediately outgrows or undershoots — too tight and the system rejects work it could handle; too loose and it melts before the limit triggers. Worse, static thresholds produce cliff behavior: the system is fully open until the instant it slams fully closed, and the synchronized rejections that follow seed retry storms that re-create the overload.

Feedback-controlled load management replaces the static number with a closed control loop. The system continuously measures a signal that reflects real load — queue wait, inflow versus outflow, high-percentile latency, error rate — compares it against a target, and smoothly adjusts admission: how many requests to accept, how long to queue them, what fraction to shed. The controller incorporates not just the current error but its history and trend, which is what produces stability — gradual corrections instead of overreactions, a dimmer switch instead of a hammer. Because the loop measures outcomes rather than assumptions, it adapts to capacity changes (hardware, workload mix, downstream slowness) without anyone retuning a config.

The pattern's lineage runs through decades of systems: TCP congestion control is a feedback loop over packet loss and round-trip time (TCP Vegas, which some modern shedders directly adapt for concurrency auto-tuning); CoDel regulates queues by measured sojourn time rather than length; Netflix's adaptive concurrency limits compute limits from observed latency gradients; and Uber's Cinnamon applies a PID controller to request shedding, with a pluggable-signal design that feeds heterogeneous overload indicators — local concurrency, memory pressure, replication lag — into one unified decision loop, eliminating the split-brain behavior of independent per-signal limiters.

The deeper principle has two halves. First: measure the system you have, not the system you provisioned. Second: when several controllers act on the same resource, unify them — competing feedback loops fight, oscillate, and make globally incoherent decisions; one loop consuming many signals makes one coherent decision.

When it applies

  • Admission control and load shedding for services whose capacity varies with workload mix, hardware, or downstream health — anywhere a static threshold needs perpetual retuning.
  • Replacing cliff-shaped overload behavior (fully open → fully closed) with smooth degradation, especially where synchronized rejections trigger retry storms.
  • Auto-tuning concurrency limits, queue timeouts, or rate caps from observed latency and error signals rather than provisioned estimates.
  • Unifying multiple overload signals (local resource pressure, replication lag, tenant skew) into a single admission decision, where independent per-signal limiters would conflict.

Tradeoffs

  • The tuning burden moves rather than vanishes. Per-service static configuration disappears, but the controller itself must be made stable — gains, targets, and damping that neither oscillate nor respond sluggishly — and that calibration is paid at the platform level, often through several iterations before the loop behaves across diverse workloads.
  • Feedback loops are harder to reason about than thresholds. 'Why was this request shed?' has a one-line answer under a static limit and a controller-state answer under feedback control; operators need visibility into the loop's inputs and decisions or they will distrust it during incidents.
  • Signals must be normalized to compose. A loop consuming heterogeneous inputs (latency, lag, bytes, counts) needs them on scales that combine meaningfully; a badly scaled signal can dominate or destabilize the decision.
  • Control latency is real. A loop that measures outcomes necessarily reacts after the fact; sudden load steps are absorbed by the loop's response time, which is why feedback control complements — rather than replaces — client-side discipline like backoff and jitter.

Seen in

  • Uber EngineeringApr 20, 2026

    From Static Rate Limiting to Intelligent Load Management

    Cinnamon is the worked instance: a PID loop over queue and latency signals smoothly adjusts admission instead of snapping a static threshold, and the BYOS design extends the same loop to any normalized overload signal — commit lag, write bytes, memory pressure — unifying what were previously competing per-signal limiters into one coherent decision path.

  • Netflix Technology BlogJun 25, 2024

    Service-Level Prioritized Load Shedding

    Netflix's adaptive concurrency limits are a textbook feedback loop: the limit is adjusted each sample from the latency gradient (RTTnoload/RTTactual) via newLimit = currentLimit × gradient + queueSize, borrowed directly from TCP congestion control, with no manual tuning or central coordination. The prioritized shedding layered on top decides which traffic to drop once that adaptively-discovered limit is reached. Seen now at both Uber (Cinnamon's PID loop over queue/latency signals) and Netflix (gradient-based concurrency limits over RTT) — two instantiations of measure-the-system-you-have rather than provision-a-static-threshold.