Problem · seen at 5 companies

Single-cluster scaling ceiling

Everything runs on one cluster, so it can only grow so far — and when it stumbles, everything stumbles with it.

The wall

A single database cluster becomes both a capacity ceiling and a shared-fate domain: it can only be scaled by ever-larger machines, and any incident on it degrades everything built on top — so relief means moving whole tables or domains onto separate clusters. 5 teams hit this wall; the breakdowns below are the evidence.

Same wall, 5 systems

GitLab

GitLab Blog (Engineering)

Every write went to one 96-vCPU primary, and there was no larger machine. Read the breakdown →

GitHub

The GitHub Blog

A decade of code assumed one shared database - the real ceiling was the cluster. Read the breakdown →

Google

Google Cloud Blog

GFS hit its metadata ceiling; Colossus rebuilt the metadata service on Bigtable. Read the breakdown →

Slack

Slack Engineering

Slack's biggest customers outgrew the largest shard a single host could sustain. Read the breakdown →

Airbnb

Airbnb Engineering

One monolithic database carried a third of all writes, and growth would overwhelm it within months. Read the breakdown →

Patterns in this class

Every breakdown

GitLab Blog (Engineering) · Aug 4, 2022Half the Writes Must Go: GitLab's Database Decomposition The GitHub Blog · Sep 27, 2021Virtual Before Physical: Partitioning GitHub's Relational Databases Google Cloud Blog · Apr 19, 2021The Ceiling Was Metadata: A Peek at Google's Colossus Slack Engineering · Dec 1, 2020The Largest Shard Money Could Buy: Slack's Road to Vitess Airbnb Engineering · Oct 6, 2015Zero Migration Code: Splitting Airbnb's Main Database in Two Weeks