The Ceiling Was Metadata: A Peek at Google's Colossus
Nearly everything Google serves (YouTube, Gmail, Drive, Search, and every Google Cloud storage product) runs on one file system, Colossus. It exists because its predecessor hit a wall. The old Google File System (GFS) kept a bookkeeping layer, called metadata, that tracked what files exist and where their pieces live. That layer stopped scaling when Google pushed it to handle Search. Colossus rebuilt the metadata to scale out: parallel servers called Curators handle operations like creating a file, and they store the metadata in Bigtable, a database built to grow. The post credits that one change with scaling Colossus more than 100x past the largest GFS clusters. Around that core sits a smart client library, file servers that stream data straight to applications, and background workers (Custodians) that keep storage healthy. One cluster reaches exabytes across tens of thousands of machines: a shared pool where live serving and batch jobs fill each other's idle gaps, with the most-used data kept on a little fast flash storage.
Run the cluster: grow the file count until a single metadata master hits the GFS wall, then switch to distributed Curators (metadata on Bigtable) and watch the ceiling move. Then run the efficiency doctrine: pool each workload's separate storage into one shared pool and price the savings, then size the flash (fast storage) tier: too little drowns the disks, too much burns money, and 'just enough' is the rule.
Problem
The post opens with the fact that does most of the work: the storage system under Google Cloud is the same one under YouTube, Drive, Gmail, Search, and Maps. All of it rests on three shared building blocks:
- Colossus, the cluster-level file system and successor to GFS.
- Spanner, Google's globally consistent database, which for the storage services keeps track of access permissions and where data lives.
- Borg, the job scheduler that launches everything from compute to storage, and a major influence on Kubernetes.
Every Google Cloud storage product, from Cloud Storage to Firestore to Cloud SQL, is a layer on top of these three.
The problem Colossus was built to solve is its origin: GFS could not scale its metadata to what Search needed. A file system's metadata is consulted on every operation (creating a file, opening it, finding which servers hold its pieces), and in GFS that bookkeeping lived in a design with a hard ceiling. When the metadata tops out, the whole cluster tops out: it doesn't matter how many disks or file servers you can rack if the layer that knows where everything lives cannot grow. That is the shape the whole problem takes: the cluster's real ceiling turns out to be one specific, load-bearing part, and here that part was the metadata service itself.
There is a second, quieter problem the post lays out: hardware variety and constant failure. Google's data centers mix spinning disk and fast flash storage in many sizes and generations, and applications differ widely in how much durability, availability, and speed they need. At Google's scale hardware is failing basically all the time, not because it is unreliable, but because there is so much of it. A file system serving everything from boot disks to archival analytics has to absorb that variety and that failure rate without passing either on to applications.
Solution
Colossus's fix has two sides: a metadata service that scales, and an architecture that keeps everything else off the metadata path. Start with the metadata service, the part that coordinates the file system rather than storing the file data. For metadata operations like creating a file, an application talks directly to parallel servers called Curators, and you scale the service by simply running more Curators. The metadata they manage is stored in Bigtable, a Google database built to scale out.
This is the heart of why Colossus scales where GFS did not. GFS kept its metadata in a single service that could only grow as large as one machine allowed; once Search pushed past that, there was nowhere left to go. Bigtable spreads the same metadata across many machines and grows by adding more of them, so the bookkeeping is no longer trapped on one box. Storing file metadata in Bigtable is what let Colossus scale up by over 100x beyond the largest GFS clusters.
Around that service, the rest of the design keeps data off the metadata path:
- The client library is the piece of Colossus code that each application (say, the servers behind Gmail or Drive) links in to talk to the system. It is described as probably the most complex part of the whole system. Heavy functions like software RAID (spreading a file's data across many drives so it survives a drive failure) live here, and each application picks encodings that trade off performance and cost for its workload.
- File data does not pass through the metadata service at all. It flows directly between the application and the D servers, the machines that actually hold the disks, cutting network hops.
- Background workers called Custodians keep the system healthy, balancing disk space and rebuilding lost copies of data continuously, rather than scrambling during a crisis.
The result is a single cluster that scales to exabytes across tens of thousands of machines, and the post is explicit about what that size buys: disaggregation. Instead of giving each workload its own separate storage (its own silo), everything shares one big pool, and each workload is made to feel like it still has its own private file system. The reason that saves money is timing. Size the shared pool for the busy peaks of the latency-critical work, like serving a YouTube video, and let batch jobs (big background analytics that can run anytime) fill the quiet stretches when interactive demand is low. Separate per-workload storage cannot do this, because each silo has to be sized for its own peak and sits idle the rest of the time.
Applications don't reason about hardware at all. They state how much speed, availability, and durability they need, and buy storage as plain units, while Colossus routes around the constant hardware failures and repairs itself in the background.
Placement is where the efficiency rule gets concrete, and it comes down to a mix of two kinds of storage: flash (fast solid-state storage, like an SSD) and disk (slower, cheaper spinning drives). The hottest data, whatever is being read most, goes on flash so it stays fast. The trick is the amount: buy just enough flash to take the heavy read demand off the disks, and just enough disk for the bulk capacity, so neither kind of drive is wasted. There is a matching rule inside the disk tier. New data, which tends to be the hottest, is spread across all the drives so reads are shared. As it ages and cools, it is moved onto bigger drives, keeping every disk full and busy. A companion post from 2025 goes deeper on placement; this dissection covers only the 2021 architecture piece.
Tradeoffs
- Rebuilding the metadata layer on a scalable database moves the ceiling rather than deleting it. Curators scale out and Bigtable scales with data volume, so the wall that stopped GFS is gone. But every operation still goes through the metadata service, so the system's ceiling is now wherever Bigtable's is. The honest reading of '100x over the largest GFS clusters' is that you inherit the ceiling of whatever you build the bookkeeping on; Google picked a foundation whose ceiling is far away, not absent.
- Putting the intelligence in the client library (the Colossus code each application runs inside its own servers) trades simplicity for a wider spread of complexity. The post calls the client probably the most complex part of the whole file system: software RAID, per-workload encodings. That keeps the servers that store data simple and the data paths direct, but it means every application runs the hardest code in the system, and changing that code means updating every application. It is a deliberate choice to put the complexity where it can be tuned per workload, paid for in coordination.
- Sending data straight from the application to the D servers (the machines that hold the disks) removes hops and bottlenecks, and also removes a natural place to enforce policy. Because data streams directly between them, the metadata service can never be a bandwidth bottleneck. But the traffic-control jobs a middleman in the data path would normally handle (limiting how fast each application can go, tracking usage, deciding who gets admitted) now have to live in the client library or the D servers themselves. This is not about metadata; it is about who polices the flow of the actual data. The design bets the client is the right home for that control, matching where it already put the complexity.
- The shared pool's efficiency comes from deliberately tying workloads together. Sizing for the peaks and backfilling with batch work is only possible because YouTube serving and Ads batch jobs sit in one pool, the opposite of the usual isolate-everything instinct. The catch: because everyone really shares one pool, Colossus has to keep up the appearance of a separate, private file system for every workload, all the time. That illusion is now ongoing work the platform can never stop, for every kind of workload at once.
- 'Just enough flash' is an efficiency rule with a moving target. Buying just enough fast flash storage to take the heavy read demand off the cheaper disks gets the most out of each kind of drive. But the right mix depends on how heavily each workload is being used, and that changes as products and usage change, so it is something you tune continuously, not a one-time purchase. The same idea runs inside the disk tier: new, hot data is spread across all drives, and as it cools it moves to bigger drives.
- Told from the far side, this story teaches the architecture but hides the road to it. Google's post is a short, six-minute overview in a promotional register, developed from a conference talk. What it leaves out is the hard part: the years between hitting GFS's ceiling and trusting the replacement, the migration itself, and the failures along the way. What you can learn here stops where the post does; the lived experience of getting from the old system to the new one is simply not in it.
Patterns in this article
- Distributed Metadata Model
This pattern comes straight from the post's own phrase: Colossus 'introduced a distributed metadata model.' Parallel Curators handle the metadata operations, and the file system's metadata is stored in Bigtable, a database that scales out. The general move it names: when a system's ceiling is its bookkeeping layer, don't keep tuning the custom design. Rehost the metadata on a database built to scale out, and let the metadata service scale like an ordinary stateless service. The source credits this with 100x the scale of the largest GFS clusters.
- Shared-Pool Multiplexing
This pattern comes from the part of the post about disaggregation: instead of siloing storage per workload, one exabyte-scale pool is shared by live serving and batch work, with each workload feeling like it has its own file system. The pool is sized for the interactive peaks, and batch jobs fill the quiet stretches. The same idea runs inside the hardware tiers: buy just enough fast flash storage to take heavy read demand off the disks, spread new hot data across all drives, and move cooling data to larger drives. It was a new pattern for the catalog; no existing one fit.
Also solving this
Other systems in behindscale's Single-cluster scaling ceiling class: