Simulated Policy Selection
throughput
Definition
When a system must choose among competing policies — what to cache, where to place data, how long to retain it, when to admit or evict — and the right answer varies by workload, the common failure is picking one policy in a design review and letting it govern everything. The pattern replaces the argument with an experiment that never ends: run cheap online simulations of every candidate policy against the live, observed workload, segment by segment, and let each segment be governed by whichever policy currently wins its simulation. Google's L4 is the reference form: applications pass features with each new file, L4 segregates files into categories, observes each category's real I/O over time, and continuously simulates the candidate placements — SSD for one hour, for two hours, not at all — with the winner governing that category's new files and the losers running on as counterfactuals, ready to take over when the workload drifts.
The pattern pays a second dividend that is easy to miss: the same simulations answer sizing questions. Because the machinery can already evaluate 'what would this policy have done,' it can equally evaluate 'what would I have done with more or less capacity' — turning placement simulation into a capacity oracle that predicts, per unit of additional fast storage, how much load could be offloaded from the slow tier. In L4's case those signals drive SSD hardware purchases and tell planners where shifting capacity between applications buys the most efficiency: the component that places files also sizes the fleet.
Boundary against neighbors: this is not A/B testing — no user-visible variation is served; the losing policies run only in simulation against the same observed trace. And it is not offline modeling — the simulations run continuously against live traffic, so a policy that stops winning stops governing. The pattern applies when a policy's counterfactual cost can be computed cheaply from an observed trace; if evaluating a candidate requires actually serving traffic under it, you are in experiment-design territory, not simulation territory.
When it applies
- The right policy genuinely varies by workload segment — a single static choice (one TTL, one placement rule, one admission threshold) is measurably wrong for some large class of traffic
- A candidate policy's outcome can be computed cheaply from the observed request trace alone, without serving real traffic under it — placement, admission, eviction, and retention decisions usually qualify
- Capacity or sizing questions recur ('how much fast tier should we buy, and for whom') and would otherwise be answered by static guesswork rather than by the same models that make the runtime decisions
Tradeoffs
- Decision quality is bounded by simulation fidelity: the simulated cost model must track real device and system behavior, or the winning policy is only winning on paper — the simulation becomes a confident source of wrong answers
- Segment granularity is itself a bet: coarse segments learn quickly from abundant data but average over hidden sub-behaviors, while fine segments fit tightly but starve each simulation of observations — and the segmentation is only as honest as the features supplied to build it
- Responsiveness is bounded by the observation window: a segment whose behavior shifts is governed by the policy its history earned until the simulations re-decide, so the pattern trades instant adaptation for decisions that are continuously validated rather than guessed
Seen in
- Google Cloud BlogMar 26, 2025
Betting on a File's Future: How Colossus Picks What Lives on SSD
Minted here. L4 never argues about placement policy in the abstract: candidate policies — SSD for an hour, two hours, not at all — run continuously as online simulations against each category's observed I/O, the winner governs that category's new files, and the losers keep running as counterfactuals ready to take over when the workload drifts. The second dividend is the capacity oracle: simulating under more or less SSD converts the same machinery into purchase-driving predictions of HDD I/O offload per unit of flash.