Service-Level Prioritized Load Shedding

PlayAPI is a critical control-plane service behind Netflix playback: when a device starts a stream it makes the manifest and license requests PlayAPI serves. But not all of PlayAPI's traffic is equally important. User-initiated requests fire when someone presses play and directly gate whether the video starts. Pre-fetch requests fire optimistically while a user merely browses, to warm the path in case they press play later — a pre-fetch failure costs a little extra startup latency, not a failed playback.

To survive traffic spikes, high backend latency, or an under-scaled dependency, PlayAPI used a single concurrency limiter that throttled requests indiscriminately. That had two bad consequences. A spike in pre-fetch traffic — which is high-volume and bursty — reduced availability for user-initiated requests, even though the user-facing work was the part that mattered. And when backend latency rose, the limiter shed both request types equally, sacrificing playback starts at moments when the system still had enough capacity to serve all the user-initiated work if only it had spent its budget wisely.

The obvious fix — shard critical and non-critical requests onto separate clusters — works and adds real failure isolation, but it carries a permanent compute-cost premium (two clusters sized for peak instead of one) plus the operational overhead of standing up CI/CD, autoscaling, metrics, and alerts for the new cluster. Netflix wanted the isolation without paying for two clusters.

This is the latest step in an arc. In 2018 Netflix open-sourced adaptive concurrency limits, which solved how much load a service should admit by discovering the limit automatically (see https://netflixtechblog.medium.com/performance-under-load-3e6fa9a60581). In 2020 it added prioritized load shedding at the Zuul API gateway, deciding which traffic to drop at the edge (https://medium.com/netflix-techblog/keeping-netflix-reliable-using-prioritized-load-shedding-6cc827b02f94). The 2024 work pushes that same which-to-drop prioritization down into the individual service — where teams own finer-grained logic, where it works for backend-to-backend calls that never touch the edge gateway, and where one cluster can safely mix request types it used to isolate physically.

PlayAPI keeps a single cluster serving both request types but adds a partitioned concurrency limiter, built on the partitioning feature of Netflix's open-source concurrency-limits library. The limiter defines two partitions: a user-initiated partition guaranteed 100% throughput, and a pre-fetch partition that may use only excess capacity. The partition sizes adjust dynamically, so user-initiated requests can 'steal' pre-fetch capacity whenever they need it — application-level isolation that gets the failure-isolation benefit of separate clusters at the compute cost of one.

The limiter is implemented as a pre-processing Servlet filter that reads a request's criticality from an HTTP header (X-Netflix.Request-Name) rather than parsing the request body. That placement is deliberate: it lets the limiter reject low-priority requests using minimal CPU, so the shedding mechanism never becomes the bottleneck it exists to prevent. Crucially, in steady state there is no throttling at all — prioritization has zero effect until the server actually hits its concurrency limit and must reject something. The mechanism is dormant until the moment of contention, then it shapes which traffic survives.

Underneath sits the 2018 substrate. Adaptive concurrency limits discover how many in-flight requests a server can handle before latency degrades, with no manual tuning and no central coordination. Borrowing from TCP congestion control, the algorithm tracks the gradient between best-case and current latency — gradient = RTTnoload / RTTactual — and adjusts the limit each sample by newLimit = currentLimit × gradient + queueSize (queueSize defaulting to the square root of the current limit, which grows fast at small limits and stabilizes at large ones). A gradient of 1 means no queue and room to grow; below 1 means a queue is forming and the limit should shrink. The result is the characteristic probe-and-back-off saw-tooth, a limit that tracks real capacity as topology shifts. Prioritized shedding decides which requests to drop once that adaptive limit is reached.

Netflix then generalized the approach beyond PlayAPI's two-partition case into an internal library with four standardized priority buckets, modeled on Linux tc-prio levels: CRITICAL (core functionality, shed only in complete failure), DEGRADED (affects experience, shed progressively as load rises), BEST_EFFORT (invisible to the user, shed readily), and BULK (background work, routinely shed). Services map incoming requests into a bucket by inspecting headers or request attributes.

The generalized framework sheds on signals beyond concurrency. Because most Netflix services autoscale on CPU, the framework ties shedding to CPU utilization — but with a deliberate ordering: it only begins shedding after the autoscale target is crossed, so the scaling signal is preserved, and as load climbs further it sheds progressively more important buckets in a staggered cascade (BULK fully shed before BEST_EFFORT begins, and so on up to CRITICAL, which sheds only past a much higher threshold). For IO-bound services that CPU doesn't describe, the framework adds latency-SLO-based utilization: a service shedding when its own per-endpoint latency targets are breached, or when a backing datastore (via Netflix's Data Gateway) reports its storage-latency SLO is saturating. That last form lets a service protect a downstream store it's overloading before the store fails — shedding low-priority reads to preserve critical writes, while still accepting as much work as the backend takes at low latency.

The design also names two anti-patterns it avoids: no shedding at all (latency rises for everyone, instances go unhealthy, and a death spiral can take down the fleet before autoscaling responds), and congestive failure (shedding so aggressively, or so expensively, that successful throughput collapses below what it was before shedding started). The validation that it avoids both is that successful RPS stays flat and latency stays bounded even as offered load climbs past 6x the autoscale target.

The mechanism proved itself in a real incident. Months after deployment, an infrastructure outage disrupted streaming; once it was fixed, Android devices fired a backlog-driven 12x spike in pre-fetch requests per second, enough to have caused a second outage on systems not scaled for it. Prioritized shedding absorbed it exactly as designed: pre-fetch availability fell as low as 20% while user-initiated availability held above 99.4%, even at a moment when more than half of all requests were being throttled. The non-critical traffic absorbed the entire hit; playback never noticed.