Problem · seen at 5 companies

Priority-blind load shedding

You start dropping requests to survive the overload — and throw out your most important traffic along with the noise.

The wall

Under overload the system drops work without regard to its importance, so shedding meant to protect the service also sacrifices its most critical traffic. 5 teams hit this wall; the breakdowns below are the evidence.

Same wall, 5 systems

Uber
Uber Engineering
Priority-blind shedding drops rides at the same rate as batch jobs, protecting nothing that matters. Read the breakdown →
Netflix
Netflix Technology Blog
One concurrency limiter shed traffic blind to priority, dropping critical playback with everything else. Read the breakdown →
LinkedIn
LinkedIn Engineering
The shedder protecting members dropped member requests as readily as batch jobs. Read the breakdown →
Amazon (AWS)
Amazon Builders' Library
An overloaded server that drops its health checks convinces the fleet it's dead. Read the breakdown →
Stripe
Stripe Engineering
Undifferentiated dropping would spend charge creation to keep test traffic alive under overload. Read the breakdown →

Patterns in this class

Every breakdown