Service-Level Prioritized Load Shedding
Netflix pushed prioritized load shedding down from the API gateway into individual services, letting a single service protect its most critical traffic without splitting it onto separate clusters. Inside PlayAPI, a partitioned concurrency limiter (a cap on how many requests are processed at once, divided by priority) guarantees user-initiated playback requests full throughput while lower-priority pre-fetch requests get only leftover capacity: the isolation of separate clusters at the compute cost of one. A production incident proved it out: after an outage, a 12x pre-fetch spike was shed down to 20% availability while user-initiated playback held above 99.4%.
Inject latency into pre-fetch traffic and watch prioritized shedding protect playback while non-critical work absorbs the hit.
Problem
PlayAPI is a critical coordination service behind Netflix playback: it sets up a stream (fetching the manifest and the license) rather than delivering the video bytes itself. When a device starts a stream it makes those setup requests, and PlayAPI serves them. But not all of PlayAPI's traffic matters equally. User-initiated requests fire when someone presses play and directly decide whether the video starts. Pre-fetch requests fire optimistically while a user is just browsing, to warm the path in case they press play later, so a pre-fetch failure costs a little extra startup latency, not a failed playback.
To survive traffic spikes, high backend latency, or a dependency that hasn't scaled up enough, PlayAPI used a single concurrency limiter (a cap on how many requests it will handle at once) that throttled everything the same way. That had two bad consequences. A spike in pre-fetch traffic, which is high-volume and bursty, cut availability for user-initiated requests, even though the user-facing work was the part that mattered. And when backend latency rose, the limiter shed both request types equally, sacrificing playback starts at moments when the system still had enough capacity to serve all the user-initiated work if it had only spent its budget wisely.
The obvious fix, putting critical and non-critical requests on separate clusters, works and adds real failure isolation, but it carries a permanent compute premium (two clusters each sized for peak instead of one) plus the operational overhead of standing up CI/CD, autoscaling, metrics, and alerts for the second cluster. Netflix wanted the isolation without paying for two clusters.
This is the latest step in an arc. In 2018 Netflix open-sourced adaptive concurrency limits, which work out how much load a service should admit by discovering the limit automatically. In 2020 it added prioritized load shedding at the Zuul API gateway, deciding which traffic to drop at the edge. The 2024 work pushes that same which-to-drop prioritization down into the individual service. Three reasons drive it there: teams own finer-grained logic at the service level, the protection now covers backend-to-backend calls that never pass through the edge gateway, and a single cluster can safely mix request types that used to be split onto separate hardware.
Solution
PlayAPI keeps a single cluster serving both request types but adds a partitioned concurrency limiter, built on the partitioning feature of Netflix's open-source concurrency-limits library. The limiter defines two partitions: a user-initiated partition guaranteed 100% throughput, and a pre-fetch partition that may use only excess capacity. The partition sizes adjust on the fly, so user-initiated requests can 'steal' pre-fetch capacity whenever they need it. That gets the failure-isolation benefit of separate clusters at the compute cost of one.
The limiter is implemented as a pre-processing Servlet filter that reads a request's criticality from an HTTP header (X-Netflix.Request-Name) instead of parsing the request body. That placement is deliberate: it lets the limiter reject a low-priority request using almost no CPU, so the shedding mechanism never becomes the bottleneck it exists to prevent. And in steady state there is no throttling at all: prioritization has zero effect until the server actually hits its concurrency limit and has to reject something. The mechanism is dormant until the moment of contention, then it shapes which traffic survives.
Underneath sits the 2018 substrate: adaptive concurrency limits, which discover how many in-flight requests a server can handle before latency degrades, with no manual tuning and no central coordination. Borrowing from TCP congestion control, the algorithm watches the gap between best-case and current latency (the 'gradient', which is best-case round-trip time divided by current round-trip time) and adjusts the limit each sample: newLimit = currentLimit x gradient + queueSize. When responses are as fast as they can be the gradient is 1, meaning there's room to grow; when they slow down the gradient drops below 1 and the limit shrinks. So the limit creeps up until it senses a queue forming, then drops back and creeps up again - an up-and-down saw-tooth that keeps tracking real capacity as conditions shift. Prioritized shedding decides which requests to drop once that adaptive limit is reached.
Netflix then generalized the approach beyond PlayAPI's two-partition case into an internal library with four standard priority buckets: CRITICAL (core functionality, shed only in complete failure), DEGRADED (affects experience, shed as load rises), BEST_EFFORT (invisible to the user, shed readily), and BULK (background work, routinely shed). Services sort incoming requests into a bucket by inspecting headers or request attributes.
The generalized framework sheds on more than just concurrency. Because most Netflix services autoscale on CPU, it can shed based on CPU use, but with a deliberate ordering: it only starts shedding after the autoscale target is crossed, so the scaling signal is preserved, and as load climbs further it sheds progressively more important buckets in a staggered cascade (BULK fully shed before BEST_EFFORT begins, and so on up to CRITICAL, which sheds only past a much higher threshold). For IO-bound services that CPU doesn't describe well, it can instead shed on latency: when a service's own response times breach their targets, or when a backing datastore reports its storage-latency target is saturating. That last form lets a service protect a downstream store it is overloading before the store fails, shedding low-priority reads to preserve critical writes while still accepting as much work as the backend can take at low latency.
The design also names the two failure modes it avoids: shedding nothing (latency rises for everyone, instances go unhealthy, and a death spiral can take down the fleet before autoscaling reacts), and congestive failure, shedding so aggressively or expensively that successful throughput drops below where it started. The proof it avoids both: successful requests-per-second stays flat and latency stays bounded even as load climbs past 6x the autoscale target.
The mechanism proved itself in a real incident. Months after deployment, an infrastructure outage disrupted streaming; once it was fixed, Android devices fired a backlog-driven 12x spike in pre-fetch requests, enough to have caused a second outage on a system not scaled for it. Prioritized shedding absorbed it exactly as designed: pre-fetch availability fell as low as 20% while user-initiated availability held above 99.4%, even when more than half of all requests were being throttled. The non-critical traffic absorbed the entire hit; playback never noticed.
Tradeoffs
- The compute-versus-isolation choice is made explicit, and Netflix takes the middle path with its own cost. Physical sharding gives the strongest isolation (a pre-fetch failure can't touch a separate user-initiated cluster) but pays for two peak-sized clusters plus their operational overhead forever. Application-level partitioning gets most of the isolation at one cluster's cost, but the isolation is now logical: a bug in the limiter, the CPU cost of shedding itself, or a misclassified request can still cross the boundary that separate hardware would have enforced physically.
- Prioritization is only as good as the request taxonomy, and the taxonomy drifts. Netflix's own 2020 work flags this directly: a request deemed non-critical can quietly become critical as the product changes, and the only defense is continuous testing (live A/B experiments that throttle a priority range for a small cohort of real users to catch playback regressions). Without that ongoing validation, the shedding logic silently drops something users actually need. Classification is an editorial responsibility that never finishes.
- Tying shedding to CPU use couples it to the autoscaling signal, which forces careful threshold ordering. Shedding has to begin only after the autoscale target is crossed, or it would suppress the very CPU signal that triggers scaling, and the cluster would never grow. Get the ordering or the thresholds wrong and the mechanism either hides the need to scale or starts shedding while capacity was still available. Its correctness depends on configuration that interacts with a separate control loop.
- CPU is the wrong signal for IO-bound services, so the framework needs several utilization measures, each with its own tuning. Latency-based and storage-based shedding extend coverage to services CPU doesn't describe, but every new measure is another signal that has to be normalized, thresholded, and trusted; the datastore example sheds correctly only because someone set a datastore-latency trigger that's right for that backend. Flexibility buys coverage at the cost of per-measure calibration.
- Shedding converts one failure mode into another, gentler one, but the gentler mode still has to be handled. A shed request is a fast rejection the caller has to retry, so the system depends on clients backing off correctly (the 2020 gateway even pushes retry-timing hints to devices for exactly this reason). Without disciplined client retry behavior, mass shedding becomes a retry storm, and the load-shedding that protected the server comes back as amplified load.
- Logical prioritization on a shared cluster gives up the operational clarity of separate clusters. With physical sharding, 'how is pre-fetch doing' is a dashboard for a distinct cluster; with a partitioned limiter, critical and non-critical health are tangled in one cluster's metrics, and working out which partition is starving which needs limiter-aware observability that separate clusters would have given for free.
Patterns in this article
- Priority-Aware Load Shedding
When a service has to drop load, it drops the least important traffic first instead of throttling everything equally. Netflix does this at two grains: four priority buckets (CRITICAL, DEGRADED, BEST_EFFORT, BULK, borrowed from Linux's traffic-priority levels) shed from the bottom up, and inside PlayAPI a two-partition limiter guarantees user-initiated playback its full share while pre-fetch gets only leftover capacity. Uber solves the same problem in its storage layer with t0-t5 tiers; Netflix solves it in the service layer with request buckets. The same shed-the-lowest-value-first idea working at different layers of two different stacks is what makes it a general pattern rather than one company's trick.
- Feedback-Controlled Load Management
Netflix's adaptive concurrency limits are a feedback loop straight out of TCP congestion control: the limit is recalculated each sample from how much responses have slowed (newLimit = currentLimit x gradient + queueSize, where the gradient is best-case latency divided by current latency), with no manual tuning and no central coordinator. When things are fast the limit grows; when they slow it shrinks. The prioritized shedding sits on top and decides which requests to drop once that self-discovered limit is hit. Uber's Cinnamon runs a similar loop with a PID controller over queue and latency signals; both are the same idea of measuring the system you actually have instead of hard-coding a fixed threshold.
- Fault Isolation
This article is really about picking the cheapest isolation that's still strong enough. Separate clusters give the firmest fault isolation (a failure in one physically can't touch the other) but cost the most; the partitioned limiter trades that hardware boundary for a software one at a single cluster's cost, accepting that the boundary is now code the team has to keep correct. It's a clear worked example of fault isolation as a spectrum with a cost gradient, not an all-or-nothing choice.
Also solving this
Other systems in behindscale's Priority-blind load shedding class: