Betting on a File's Future: How Colossus Picks What Lives on SSD
Google's own line states the problem: it's hard to read at 50 TB/s if all of your data sits on slow spinning disks — and an SSD-only fleet still carries a substantial cost premium, so the bulk of Colossus's exabytes must stay on HDD while only the data that earns it gets flash. The wall is that placement is decided at the worst possible moment: when a file is created, Colossus can see only the application creating it and its name — the access pattern that would justify the choice doesn't exist yet. The answer is L4, a distributed SSD layer that works as both a read cache and a writeback cache: applications pass features about each new file, L4 groups files into categories, watches each category's real I/O, and continuously simulates competing placement policies — place on SSD for an hour, for two hours, not at all — letting the winning policy govern each category. New files from the right categories are born on SSD and migrated to HDD later; in the best case they're deleted before migration and never touch a spinning disk at all. The same simulations double as a capacity oracle that tells Google how much SSD to buy.
Run four workloads — batch intermediates that die young, chatty transaction logs, hot serving data, cold archive — through four placement regimes. Watch all-HDD saturate the spindles, all-SSD blow the budget, and a read cache help everything except the data that's hot at birth. Then turn on learned writeback placement, watch ephemeral files live and die without a single HDD operation — and shift a workload's behavior under the policy to see the bet go wrong and the simulation re-decide.
Problem
Almost everything Google runs — YouTube, Gmail, BigQuery, Cloud Storage, Spanner — sits on Colossus, the successor to the Google File System and the company's universal storage platform. A traditional Colossus filesystem is contained in a single data center: one filesystem per cluster, the internal building block of a Google Cloud zone. The scale inside that one building is the part that's hard to hold in your head: many Colossus filesystems hold multiple exabytes — a billion gigabytes each — and two hold more than 10 exabytes apiece. The largest filesystems regularly exceed read throughputs of 50 TB/s and write throughputs of 25 TB/s (Google's own comparison: enough to send more than 100 full-length 8K movies every second), and the busiest single cluster delivers over 600 million individual reads and writes per second. And the workloads sharing one filesystem want opposite things: BigQuery scans want terabytes per second of streaming throughput, Hyperdisk wants sub-millisecond reads, Gmail messages want tens of milliseconds, YouTube video and Gmail attachments tolerate seconds.
Performance at that scale is a placement problem before it is anything else — the post is blunt that it's hard to read at 50 TB/s if all of your data is located on slow disk drives. The hardware trend sharpened the question rather than answering it: SSDs — flash storage, fast but expensive — have gotten steadily more affordable, prominent enough that no storage designer would spec an HDD-only system anymore; but an SSD-only fleet still poses a substantial cost premium over a blended fleet of flash and spinning disks. So the challenge is putting the right data — the data that gets the most I/Os or needs the lowest latency — on SSD, while the bulk of the exabytes stay on cheap HDDs.
Colossus's manual options show why 'let users choose' isn't the answer. An internal user can force a file fully onto SSD through its path — the easiest option and the most expensive. A more sophisticated user can request hybrid placement, putting just one replica of the file on SSD — cheaper, but if the server holding the flash copy is unavailable, every access falls back to HDD latency. Both require the user to know the future: which files deserve flash, how much SSD quota to purchase for the workload, and when to migrate aging files back to HDD — parameters Google watched its own engineers find quite hard to decide. And underneath the human problem is the structural one, flagged in the post with its own exclamation mark: at file creation time, the moment placement is decided, Colossus can see only the application creating the file and its name. It doesn't know for sure how the file will be used — the access pattern that would justify the placement doesn't exist yet.
Solution
The bulk of Google's data takes neither manual path; it goes through L4, a distributed SSD caching layer that attacks the problem from two directions. The first is a read cache. L4 index servers maintain a distributed map of what's on flash: an application wanting data consults an index server first, and either gets pointed at SSD copies (a hit) or fetches from wherever Colossus placed the data on disk (a miss). On a miss, L4 can decide to insert — instructing an SSD storage server to pull the data across from the HDD server — and as the cache fills, it evicts older items to make room. The interesting choice is the insertion policy, because the right one varies by workload: insert data into the cache when it's written, only after the first time it's read, or only after the second read within a short period. L4 uses a machine-learning-driven algorithm (published as the CacheSack paper) to pick among those policies per workload, and the post credits this read cache with dramatically improving IOPS and throughput.
But a read cache has a structural blind spot: new data is still written to HDD first. Two important classes of data are exactly wrong for that arrangement — data that is written, read, and deleted quickly, like the intermediate results of large batch jobs, and database transaction logs and other files that take many tiny appends. Both punish spinning disks, and both are known to be hot at the moment they're born, which is precisely when the read cache can't help. For those, the right move is to write directly to SSD and skip HDD entirely — which turns the placement question from reactive to predictive, and is where the crux bites.
L4's second face answers it: as a writeback cache, L4 advises Colossus curators — the metadata service that handles file creation — on whether to put each new file on SSD, and for how long. Since the file's future is invisible at creation, L4 manufactures a proxy for it: applications pass features along with the file, such as the file type or metadata about the database column whose data it stores; L4 uses the features to segregate files into categories and observes the real I/O pattern of each category over time. Those observed patterns then drive an online simulation of competing placement policies — place this category on SSD for one hour, for two hours, or don't place it on SSD at all — and the best-performing policy in simulation becomes the governing policy for that category's new files. The bet is category-level, continuously re-validated, and grounded in what files like this one actually did.
When the curator is instructed, new files go to SSD instead of the default HDD, and after the policy's set time, any file that still exists is migrated down to HDD. When the simulation predicts a category correctly, the economics are exactly what the post's title promises: a small portion of data sits on flash and absorbs most of the reads — which skew heavily toward newly created files — before migrating to cheaper storage. In the best case, the file is deleted before its migration ever comes due, and an entire class of data lives and dies without generating a single HDD I/O.
The simulations earn their keep twice. Because L4 is already simulating placement under alternative policies, it can also simulate placement under alternative hardware — predicting what it would choose with more or less SSD available, and therefore how much HDD I/O could be offloaded per additional unit of flash. Those signals drive purchases of new SSD hardware and tell capacity planners where shifting SSD between applications would buy the most efficiency: the same machinery that places files also sizes the fleet. The results surface across Google Cloud's own products — Hyperdisk ML uses Colossus SSD to let 2,500 nodes read at 1.2 TB/s, Spanner's tiered storage puts cheap HDD and fast SSD in the same filesystem, Cloud Storage leans on L4's SSD caching to serve AI/ML I/O from the cheapest storage tier, and BigQuery rides Colossus flash for extra-large queries — while the stated goal is that none of those products' users should have to be experts on HDDs, SSDs, or Colossus at all.
Tradeoffs
- The whole scheme is a bet, and the post only narrates the winning side. Its own framing is conditional — when the simulation correctly predicts the file access patterns, a small portion of data absorbs most of the reads — which means the miss cases are real and land on both sides of the ledger: a category predicted hot that isn't wastes flash on cold data, and a category predicted cold that isn't serves hot reads from spinning disks until the observed I/O shifts the simulation's verdict. The design accepts prediction error as the price of making the decision at all; what it buys is that errors are category-scoped, continuously measured, and self-correcting rather than permanent.
- The features are supplied by the applications, so the categories are only as honest as what developers pass in. A category is the unit of both learning and placement: files sharing features share a fate, which is exactly right when the features capture the workload and exactly wrong when one category quietly contains two behaviors — the policy that wins the simulation for the blend serves neither well. Granularity is a standing tension (our reading, not the post's): coarser categories learn faster from more data but average over more variance; finer ones fit tighter but starve the simulation of observations.
- Adaptation is bounded by the observation window. The governing policy for a category reflects its observed history, so a workload that changes behavior is mis-placed for as long as it takes the simulations to re-decide — the same structural lag the read cache's ML policy selection carries. For steady workloads the lag is invisible; for a product launch or a batch pipeline that changes shape, the placement layer is briefly betting on the workload that used to exist.
- Time-boxed SSD residency is a blunt instrument by design. The policy menu the post describes is durational — SSD for one hour, two hours, or not at all — so a file that outlives its window is migrated down on schedule, and if it turns hot afterward, it is the read cache's job to notice and pull it back up. The two faces of L4 cover each other's blind spots (writeback owns hot-at-birth, the read cache owns hot-again-later), which also means the system's behavior for any one file is the composition of two learned mechanisms rather than one inspectable rule.
- Coupling capacity planning to the placement simulations doubles the payoff of being right and the cost of being systematically wrong. The signals that drive SSD purchases and cross-application capacity shifts come from the same models that place files, so a bias in the simulation doesn't just misplace some categories — it feeds a mis-sized fleet back to the planners (an implication we're drawing; the post presents only the benefit). The compensation is that these are simulations against real observed I/O, not offline guesses, and they're re-run continuously as the workloads drift.
Patterns in this article
- Simulated Policy Selection
Minted here. L4 never argues about placement policy in the abstract: candidate policies — SSD for an hour, two hours, not at all — run continuously as online simulations against each category's observed I/O, the winner governs that category's new files, and the losers keep running as counterfactuals ready to take over when the workload drifts. The second dividend is the capacity oracle: simulating under more or less SSD converts the same machinery into purchase-driving predictions of HDD I/O offload per unit of flash.
- Distributed Metadata Model
Same-company recurrence from the scaling article: the curators that scale Colossus's metadata horizontally are also the enforcement point for placement — L4 advises, but it is the curator that directs a new file to SSD and later migrates it down to HDD. The metadata layer that made exabyte filesystems possible is what makes fleet-wide placement decisions executable.
- Shared-Pool Multiplexing
Same-company recurrence, one level down the stack: the scaling article's lesson was many workloads multiplexed onto one pool of storage; this post is about the pool itself being two device tiers — a blended SSD and HDD fleet inside one filesystem — with L4 deciding, per category, which tier each file's I/O lands on. Spanner's tiered storage riding cheap HDD and fast SSD in the same filesystem is the pattern surfaced as product.