The Selfish Retry: Timeouts, Backoff, and Jitter at Amazon

Whenever one system calls another, failure can come from anywhere in the stack — servers, networks, load balancers, software, operating systems, operators. Amazon's stance is that systems which never fail cannot be built, so systems must tolerate failure without magnifying a small percentage of faults into a complete outage. The magnification is the hard part, and it arrives through the very tools meant to provide the tolerance.

Many failures first appear as slowness: requests taking longer than usual, possibly never completing. A client waiting on such a request holds its resources — memory, threads, connections — for the duration, and a server doing the same accumulates the load of requests whose callers may have long stopped caring. Unbounded waiting converts one component's slowness into everyone's resource exhaustion, which is what timeouts exist to prevent. But timeouts create the ambiguity retries must then paper over: a timed-out call may or may not have executed, and its side effects may or may not have happened.

Retries resolve the availability half of that problem brutally well — repeat the call and most transient faults disappear. The cost is the crux: retrying is selfish. Each retry asks an already-struggling dependency to spend capacity twice, or three times, for one unit of client value. If the underlying failure is overload, a fleet of retrying clients converts a brownout into a sustained storm, and the system can stay down under retry pressure long after the original cause has passed. Backoff — spacing retries out with exponentially growing waits — keeps the load on the backend more even, but has its own failure mode: correlation. Clients that failed at the same moment back off by the same schedule and return at the same moment, re-creating contention in synchronized waves. And traffic at Amazon doesn't arrive smoothly to begin with — client behavior, failure recovery, and even ordinary cron jobs produce large bursts, so anything that synchronizes clients further is manufacturing the next spike.

Timeouts come first, and Amazon's practice is uncompromising: set one on any remote call — and on any call across processes even on the same server — covering both connection and request phases. The value is chosen deliberately rather than folklorically, from the dependency's measured latency distribution, by picking an acceptable rate of false timeouts: set the bound where only a chosen small fraction of genuinely healthy requests would be cut off. Too generous and resources pool behind slow calls; too aggressive and the timeout itself manufactures retries.

Retries are then applied with the article's defining caution. Because retries amplify load on a failing dependency, Amazon retries only when there is evidence the dependency is healthy, and stops retrying when retries are not improving availability — spending capacity on retries that cannot succeed only deepens an overload. The enforcement mechanism is mechanical rather than aspirational: clients throttle their own retries with a local token bucket, retrying freely while tokens remain and at a fixed, bounded rate once they are exhausted. That behavior shipped inside the AWS SDK in 2016, so every customer using the SDK carries the protection by default — the platform makes the safe behavior the ambient one.

Whether to retry at all is an API-contract question before it is a client question. A timeout does not mean the side effects didn't happen, so APIs with side effects are unsafe to retry unless they provide idempotency — the guarantee that effects occur once no matter how many attempts arrive. Read-only APIs are typically idempotent already; resource-creating APIs may not be, which is why EC2's RunInstances offers an explicit token-based mechanism: the client supplies a token, retries carry the same token, and the second launch request becomes a harmless echo of the first rather than a second fleet of instances.

Backoff spreads retries in time — capped exponential growth between attempts keeps backend load even — but backoff alone leaves the correlation problem: clients that failed together return together. Jitter, a randomized component in the wait, spreads the synchronized wave into an approximately constant rate, and the article points to Amazon's analysis of jitter strategies for how much to add and how. Then comes the observation that elevates the article beyond retry hygiene: jitter isn't only for retries. Traffic to Amazon services spikes in bursts short enough to hide inside aggregated metrics, and much of that burstiness is self-inflicted — clients calling on regular intervals line up with each other, cron jobs cluster at the top of the hour and the first seconds after midnight. On systems like EBS and Lambda, deliberately jittering clients' periodic workloads let the same work complete with less server capacity. Amazon therefore considers adding jitter to all timers, periodic jobs, and delayed work — chosen consistently per host rather than freshly random, so that under overload the pattern remains legible to a debugging human.