Who Gets Dropped: Hodor and Overload Protection at LinkedIn

LinkedIn launched over eighteen years before these posts as a single monolithic Java web application; by the time Hodor was built, the site ran on well over 1,000 separate microservices on the JVM — the Java Virtual Machine, the runtime all of them share — working in tandem toward a 99.9% availability goal. At that scale, overload is a routine failure mode: a service pushed past its capacity stops serving traffic at reasonable latency, and left to its own devices the condition can spread until it threatens the whole site. LinkedIn had used the standard mitigations for years, and the 2022 post prices each one honestly: overprovisioning handles the expected peak but raises hardware costs; circuit breakers protect the client from a struggling downstream but do nothing for the service actually under load; and capacity numbers approximated by pushing a service to its bending point are rough, static guides that can't account for unexpected failures. What LinkedIn did not have was a standard server-side answer — a way for any service, under any overload, to protect itself.

The constraint that shapes everything is heterogeneity at fleet scale. The services have disparate workloads, execution models, and traffic patterns, so the framework had to detect a wide range of overload types: physical resource exhaustion — CPU, memory, network and disk I/O — and virtual resource exhaustion, such as execution threads, pooled database connections, and semaphore permits. The virtual kind can arrive without any increase in incoming traffic at all, because a slow downstream raises the number of requests concurrently in flight upstream. And whatever the mechanism, it had to work out of the box: one of Hodor's four stated goals is that no service owner or SRE ever tunes the algorithm. A protection scheme that needs per-service calibration will not cover a thousand services; the fleet-wide default is the whole point — and it is also the trap.

The trap is stated in the framework's own goal ranking. The fourth goal — 'the most important goal' — is that Hodor produce a net positive member impact: it exists to improve the member experience, and a net negative member experience caused by Hodor is unacceptable. But load shedding is refusal, and a shedder that runs everywhere by default, with no tuning and no knowledge of what it is refusing, has two ways to hurt the members it protects. It can fire when nothing is wrong — which is why the detectors optimize for precision over recall, and why Hodor's own overhead must be negligible, since it runs on every service at all times. And it can drop the wrong traffic: LinkedIn's services serve members actively using the platform alongside offline systems harvesting data, and the post is blunt that requests from members are more important than requests from offline jobs. A concurrency cap that rejects overflow indiscriminately spends its protection on exactly the requests the whole framework exists to keep whole.

Hodor's sensing is built on one transferable idea: measure overload from inside the process, as the application experiences it, rather than from outside it. The CPU detector is a background heartbeat thread that sleeps for a fixed interval and, on waking, compares how long it actually slept with how long it asked to sleep. The 2022 post gives a concrete example of the shape: sleep for 10ms; if the 99th percentile of a second's worth of samples exceeds 55ms, that window is in violation; eight consecutive violating windows mean the service is CPU-overloaded — thresholds arrived at through synthetic testing and production data. Measuring the ability to get CPU time inside the JVM, instead of watching OS-level CPU percentages, makes the signal architecture-agnostic — nothing changes moving between bare metal, containers, and VMs — and it sees what OS metrics miss: safepoint pauses, the moments the JVM stops all application work for garbage collection, programmatic thread dumps, or revoking locks, which hurt requests without necessarily showing in CPU graphs.

Two more detectors extend the net. The garbage-collection detector computes, on each GC event, an amortized percentage of time spent in GC — the GC overhead — and divides the percentage range into tiers, each with a violation period that shrinks as the tier rises: the more severe the GC activity, the less time Hodor waits before calling it an overload. The threadpool detector covers the virtual side, and its design choice is the post's sharpest lesson in metric selection. Backpressure — a slow downstream causing requests to accumulate upstream — typically exhausts a virtual resource like a threadpool before any physical limit is hit, and the obvious signal is queue length. The team dismissed it: queue length is difficult to correlate with any business-impact indicator, and the length at which a given service is in trouble varies with its design. Queue wait time survived instead — experimentation showed a wait-time inflection point past which a service does not return — which means an upstream service can sense a downstream's distress from its own queue, knowing nothing about the downstream at all, and shed at the point where relief actually helps. The post generalizes the idea explicitly: observing wait times to detect virtual resource overloads applies widely.

Behind the detectors sits a filter that encodes the framework's epistemics. Latency was never viable as a primary detector — not every latency increase is an overload — but every physical overload eventually surfaces as a latency increase, so latency works as confirmation. After the broad rollout, services with suboptimally tuned garbage collection produced micro stop-the-world pauses that starved the heartbeat thread and fired CPU overloads with no discernible impact on business metrics. The latency confirmation filter suppresses those: two sliding windows of very different sizes compute a slow and a fast moving average of latency, and a firing only counts when the fast average crosses the slow one shifted up by a factor. The post is candid that the crossover algorithm is far from perfect as standalone anomaly detection; as a confirmation gate on the other detectors, it is exactly enough.

Remediation began as an adaptive concurrency limit — and the adaptivity, not the limit, is the mechanism. When a detector fires, the shedder derives an initial cap on concurrent requests from the concurrency levels observed leading up to the overload, holds the service at an equilibrium it can survive, then periodically probes upward to see whether more traffic fits. A probe that re-fires the detectors is rolled back, and an exponential backoff stretches the time until the next attempt. The limit is deliberately recalculated at every new overload rather than remembered, because the last incident's number describes the last incident: the traffic pattern may differ — a surge in organic traffic one time, an offline or nearline job arriving the next — and so may the nature of the overload, since backpressure from a downstream changes capacity differently than additional organic traffic does. Concurrency beat percentage-based throttling in dark canary testing — parallel copies of services fed real production traffic — for a structural reason: a concurrency cap self-adjusts as traffic moves, automatically dropping more when traffic rises and nothing once it falls back under the limit, where a fixed percentage over-drops or under-drops.

Traffic tiering is where the shedder learns who it is dropping, and it is the piece the crux turns on. Every request carries one of three tiers: optional — droppable with no user impact at all, where most offline and nearline traffic lands; degradable — minimal user impact, the tier product owners assign to degradable features on a page, and the promotion path for offline traffic that is time-sensitive or business-critical; and non-degradable — online member requests that cannot be degraded. The numeric tier IDs (1000, 5000, 10000) are spaced to leave room for finer priorities later; the first service in the data center to handle a request stamps a tier if none is set, and the tier then rides the entire distributed call tree. The shedder drops from the bottom of the ladder up — optional before degradable before non-degradable — and within a tier, members are hashed into buckets and whole buckets are dropped, concentrating degradation on a consistent few call trees instead of spraying it randomly across everyone; if shedding a few buckets or a lower tier isn't enough, the threshold walks up until the system stabilizes or recovers fully. Integrating this broke the original load metric: dropped requests contribute nothing meaningful to a concurrency histogram, so tier-aware shedding forced a switch to a rate-based limit — the same algorithm with throughput as the load measure — backed by per-endpoint histograms of the incoming tier and bucket distribution. That histogram is what makes the ladder honest: the shedder knows, before it drops, whether any lower-priority traffic is even present, and an endpoint receiving only degradable traffic is shed accordingly.

The last line of member protection is what happens to a dropped request. Because Hodor rejects in the filter chain, before any application logic runs, a rejection — a 503, in LinkedIn's own Rest.li service framework — is safe for the client to retry on another instance regardless of the request's idempotency. Blind retries would trade one overload for a retry storm, so budgets bound them on both sides, client and server, in a design the post credits to Google's SRE book; an exhausted server-side budget is itself a signal that the problem is widespread, at which point the server stops instructing clients to retry and accepts failed requests to protect goodput for the traffic still being served. Rollout carried the same do-no-harm posture as the design: every service ran detectors in monitoring mode over at least a week of data, including load tests, with firings correlated against performance metrics before shedding was ever enabled. The flagship Voyager services went first, replacing a hand-tuned fixed-concurrency scheme, and Hodor promptly caught overloads the manual thresholds had missed. The monitoring phase doubled as a fleet-wide profiler, surfacing GC problems, periodic application-triggered thread dumps stalling safepoints, and GC-log writes blocked on other applications' disk I/O — problems owners didn't know they had. Hodor now runs on more than 1,000 microservices and has prevented hundreds of overloads in LinkedIn production — enough success that its signals are being considered for autoscaling and safe load testing in production.