Who Gets Dropped: Hodor and Overload Protection at LinkedIn
Hodor is LinkedIn's overload protection. It runs inside more than 1,000 of their Java services and must never make things worse for members - LinkedIn's logged-in users. Its first version capped how many requests a service handled at once and dropped the rest blindly - a member loading their feed dropped as easily as an offline bulk read. Because members matter more, the fix was to add priority: each request is given a priority - optional, degradable, or non-degradable - and the lowest is dropped first. To act early, each service watches itself for signs of overload: a thread starved of CPU, too much time lost to garbage collection, and requests waiting too long for a free worker. Adding priority then broke how the shedder measured load. It had judged load by counting the requests it was handling, but a dropped request is never handled, so that count can't show the low-priority traffic the shedder must see. It now counts how many requests of each priority arrive instead. Hodor has prevented hundreds of overloads.
Push a mix of member and offline traffic past capacity under the first, blind shedder and watch member requests get refused while offline reads sail through. Switch on priority and watch optional traffic take the whole cut instead. Push harder until even degradable traffic starts dropping, and watch the cap edge back up and then snap back when it tries too much. Finally, overload the whole cluster at once to see what the retry budget does when retrying can no longer help.
Problem
LinkedIn began as a single Java web app and, more than eighteen years on, runs on well over 1,000 separate services on the JVM (the Java runtime they all share), working together toward a 99.9% availability goal. At that scale, overload is a routine failure: a service pushed past what it can handle stops serving traffic fast enough, and left alone the trouble can spread until it threatens the whole site. LinkedIn had used the usual defences for years, and the 2022 post is honest about the limit of each:
Overprovisioning handles the expected peak but costs more hardware. Circuit breakers protect a caller from a struggling service below it, but do nothing for the service actually under load. And capacity limits found by pushing a service to its breaking point are rough, fixed guesses that can't allow for surprises.
What LinkedIn lacked was a standard answer a service could apply to itself, under any kind of overload. The hard part is how much the services vary across the fleet of more than 1,000. They run different work with different traffic, so one tool has to spot many kinds of overload. It must catch running out of a physical resource like CPU, memory, or network, and running out of a virtual one like worker threads or database connections. The virtual kind can happen with no rise in incoming traffic at all. When a service you depend on slows down, each request takes longer, so more of them are in progress at the same time, and the pool of threads fills up. On top of that, the tool has to work with no per-service tuning, one default for a thousand services, which is the whole point and also the trap.
The trap is spelled out in Hodor's own goals. Its most important goal is a net positive impact on members, and a shedder that runs everywhere by default, untuned and blind to what it refuses, has two ways to hurt the members it is meant to protect. It can fire when nothing is really wrong, which is why the detectors are tuned to stay quiet unless they are sure, and why Hodor's own cost must be tiny, since it runs on every service all the time. And it can drop the wrong traffic. LinkedIn serves live members alongside offline jobs collecting data, and members matter more, so a cap that rejects the overflow blindly spends its protection on the very requests the system exists to keep serving.
Solution
Hodor is not a separate monitor. It is a library built into each LinkedIn service, so its watching and shedding happen inside the program it protects. That choice drives everything: it measures overload as the application feels it, not from outside numbers like CPU percentage. Three detectors do the watching.
The first is a background thread that sleeps for a set time and, on waking, checks how much longer than expected it slept. If the service is starved of CPU, the thread keeps oversleeping, and a run of oversleeps means it is CPU-overloaded. Measured inside the running program, the check works the same on bare metal, in a container, or a VM. It also catches stalls that outside CPU numbers miss, like when the runtime freezes every thread to collect memory.
The second detector watches that cleanup, garbage collection, which briefly pauses real work. It tracks what share of recent time went into cleanup and sorts it into severity bands: the worse the band, the sooner Hodor calls an overload.
The third detector watches the pool of worker threads that handle requests. When a service it depends on turns slow, requests sit waiting, those threads fill up, and it runs out of them before running out of CPU or memory. The obvious signal is the length of the queue of waiting work, but that misleads: a long queue means different things for different services and is hard to tie to real user harm. They measure how long requests wait in the queue instead, because tests found a point past which a service never recovers. A caller can then sense trouble below it from its own queue wait, and shed where it helps.
Behind the three sits a latency check that rules out false alarms. Latency alone is a poor trigger, since not every slowdown is an overload, but every real overload eventually shows up as higher latency, so it works as a final confirmation. The detectors stay quiet unless sure: Hodor would rather miss a borderline case than shed on a false alarm and harm members for nothing. The check counts a firing only when a fast-moving average of latency pulls clearly above a slow-moving one, which suppresses false trips like the tiny GC pauses that fooled the CPU detector after rollout.
Once overload is confirmed, Hodor caps how many requests the service handles at once, and that cap moves. It starts from the load just before the trouble, holds it at a survivable level, then now and then tries raising the cap a little to see if more fits - a probe. If a probe brings the overload back, Hodor drops the cap to where it was and waits longer before trying again, doubling the wait each time. It computes a fresh cap for every overload, because the traffic and the kind of overload differ each time.
Priority is where the shedder learns whom it drops. Every request is given one of three priorities: optional (1000, mostly offline work), degradable (5000, minor user impact), or non-degradable (10000, member requests, never degraded). The first service to touch a request sets its priority if it has none, and the priority travels with it through the call chain. The shedder drops the lowest priority first. Within a priority, it groups users by a hash of their ID and drops whole groups, so the same few users take the hit consistently, not a random set. If that is not enough, it moves up to the next priority.
Adding priorities broke how the shedder measured load. It had counted how many requests were in flight at once, but a dropped request is never in flight, so it never shows up in that count. A shedder that chooses what to drop by priority cannot rely on a number that ignores its own drops. So Hodor switched to measuring the rate of requests per priority, which lets it check whether any lower-priority traffic is present before dropping anything.
The last protection is the dropped request itself. Because Hodor refuses it before the application code runs, it is safe to retry on another copy. Blind retries could turn one overload into a flood, so caller and server each keep a retry budget, an idea from Google's SRE book. When that budget runs out, the overload is clearly widespread, so the server stops asking for retries and lets some requests fail on purpose, keeping the traffic it can still serve. Rollout was careful too: every service ran the detectors in watch-only mode for a week before shedding turned on, which also surfaced fleet-wide tuning problems. Hodor now runs on more than 1,000 services and has prevented hundreds of overloads.
Tradeoffs
- Priority shedding moves the hard part from the shedder to the job of classifying requests correctly. The system only protects members if every request gets the right priority at the front door and keeps it through the whole chain of calls, and setting those priorities is human judgement spread across the company. Product owners decide which parts of a page are degradable, and some offline jobs are important enough to be promoted up beside member traffic. A wrong priority only bites in one moment, an overload, the one time the priority is actually used. An offline job set too high hides from the shedder while member traffic competes with it, and a member request set too low is dropped first. The priority numbers are spaced out on purpose (1000, 5000, 10000), leaving room to slot in finer levels later, which also admits that keeping every request correctly classified is work that never ends.
- Measuring from inside the process makes Hodor work the same on any hardware, but ties it to the software underneath. The rollout showed the CPU signal's strength changing with the app framework and the Java runtime. On some, the starvation showed up clearly; on others, the background thread could barely tell the service was overloaded at all. One set of thresholds had to work across all of them, and a few services could not start on the default settings until their memory cleanup was tuned or wasteful memory use was fixed. The no-tuning promise holds for the algorithm; the fleet pays for it by making the odd outlier service conform to the detector. The post counts those forced cleanup fixes as real wins in their own right.
- Choosing to avoid false alarms means accepting late and missed detections by design. The latency check exists because, after the wide rollout, tiny memory-cleanup pauses were tripping the CPU detector with no real user impact, which is unacceptable for a system whose top goal is never harming members. But a check that ignores unconfirmed alarms must wait until latency has clearly risen before acting on a real overload, and the post admits that latency rising on its own would be a poor way to detect overload. The trade is deliberate: Hodor would rather shed late than shed wrongly. That is the right call for a system on by default across the whole fleet, and the opposite of what a single hand-tuned service might choose.
- The adaptive cap deliberately remembers nothing between overloads. Working out a fresh cap every time is justified, since the traffic mix and the kind of overload differ each time, but the price is that every overload pays the cost of rediscovering the safe level. Hodor finds a survivable level by feedback, then now and then tests raising it, and those failed tests show up as small latency spikes on the protected service in LinkedIn's own before-and-after graphs. That exploration is the tax on adapting. A system that carried its last cap forward would recover faster when a new overload looked like the last one, and hurt members when it did not; Hodor chose never being confidently wrong over ever being instantly right.
- Switching to a rate limit gives up a useful property of the old count. A cap on requests in flight regulates itself: if requests get slower, more pile up at once, so the cap automatically admits fewer. A cap on request rate admits the same number per second whether each request is cheap or expensive, leaving that correction to the detectors instead of the metric. The post only explains why the switch was forced: a count of in-flight requests cannot see the traffic the shedder drops, so a priority shedder cannot be built on it. It does not mention this cost; that reading is ours. What the post does describe is the fix: Hodor keeps a live breakdown of request rates per service and per priority, which gives the rate-based shedder the awareness the old count had provided for free.
- Retrying a dropped request on another copy turns local drops into a fleet-wide rescue, but only while the overload stays local. The design is honest about the limit: when callers are retrying everywhere, the server's retry budget runs out and the server stops asking for retries. At that point members get failed requests, on purpose, because serving the traffic that can still be served beats a retry storm that finishes off the cluster. Dropping some requests to protect the rest is the framework's founding bargain, restated at the retry layer, and it gives Hodor's protection a floor. Under a truly global overload, priorities decide who fails first, budgets decide when to stop trying to save them, and some member impact is accepted as the cost of serving any members at all.
Patterns in this article
- Priority-Aware Load Shedding
LinkedIn's version is the fleet-wide default: three priorities - optional, degradable, non-degradable - set at the data-centre edge and carried through the whole chain of calls. Inside each priority, users are sorted into groups by a hash of their ID, so escalation drops whole groups at a time: the same few degrade consistently rather than everyone at random. The distinctive detail is that adding priority broke the load metric, forcing a move from capping requests in flight to capping request rate, tracked as a live breakdown per service and per priority.
- Feedback-Controlled Load Management
Here the pattern takes a test-and-back-off form: a starting cap based on the load just before the overload, a survivable level held on detector feedback, occasional upward tests, and a doubling wait when a test brings the overload back. The cap is worked out fresh at each overload, because neither the traffic mix nor the kind of overload repeats. No control-theory machinery, just the same closed loop.
- Retry Budget
The server-assisted form of the pattern: Hodor refuses a request before the application code runs, so retrying it on another copy is safe whatever it does. Both caller and server keep a retry budget, an idea from Google's SRE book, to bound the storm risk. When the server's budget runs out, that itself signals a widespread overload and switches retries off entirely, accepting failed requests to protect the traffic still being served.
Also solving this
Other systems in behindscale's Priority-blind load shedding class: