Half the Writes Must Go: GitLab's Database Decomposition
GitLab.com's monolithic Postgres held almost everything its users generated — 22TiB, excluding git data — behind a Patroni replica pool and PGBouncer connection pooling that scaled reads for years and could never touch the real bottleneck: every write goes to the primary, the primary was a single 96-vCPU VM, and no larger machine was coming. Dylan Griffith's three-part series narrates the yearlong answer. Horizontal sharding by top-level namespace was explored first and deferred — the application was never designed with tenancy boundaries — so the shorter-term move was functional decomposition, and the split line was chosen by measurement: CI tables carried roughly 49% of all writes and 36% of the data, making a CI/Main split the optimal halving. Getting there meant governing a moving target with a violation ratchet (detect cross-joins and cross-database transactions, allowlist the existing offenders, fail CI on any new one), rebuilding referential integrity as loose foreign keys (on-delete triggers feeding a queue drained by Sidekiq), mirroring two stubborn tables with periodic consistency checking, and a seven-phase rollout whose key insight — run the application as if it had two databases while both connections point at one — made the final production change a trivial host reconfiguration. The team evaluated a near-zero-downtime cutover and deliberately declined it for a two-hour window with a clean rollback; after seven staging rehearsals, production took 93 minutes. Vacuum saturation fell from 80–100% peaks to about 15%, and Sidekiq's average query duration improved five-fold once connection limits could finally be raised.
Add replicas to a write-bound primary and learn why they don't help; ask for a bigger machine than 96 vCPUs and find the store empty. Run the write analysis, arm the violation ratchet and watch it block a fresh cross-join, replace the foreign keys with queues, run the app as-if-two-databases on one host — then rehearse seven times, decline the zero-downtime plan for reasons you can read, and cut over in 93 minutes.
Problem
The pre-decomposition architecture was already a well-scaled monolith: one Postgres database holding almost all data generated by GitLab.com users (git data aside), fronted by Patroni managing a pool of read-only replicas and PGBouncer pooling connections from the application fleet. The post is direct about what that bought and what it couldn't: these approaches only got so far, and would never fix the scaling bottleneck of the number of writes — because all writes go to the primary. The primary was a single VM at 96 vCPUs, already near the limits of what one machine could do, and the escalation path was closing: continually vertical-scaling would eventually not be possible, and even a hypothetically infinite machine leaves a 22TiB database that grows harder to manage and had become a common source of incidents.
The database sharding team formed in early 2021 aimed first at the long-term answer: horizontal sharding by top-level namespace. That exploration hit the application's history — GitLab was never designed with strict tenancy boundaries around namespaces, so the complicated problems stacked up. The team's judgment is carefully stated: sharding will ultimately be a good way to split and scale, but the scaling problem needed a shorter-term solution. That reframing — pick the reachable move, keep the ultimate move on the map — led to functional decomposition: extract a set of tables into a separate database, provided a set could be found with loose coupling to the rest, knowing every join across the line would have to die.
The split line came from measurement. An analysis of where data lived and where writes landed produced the deciding table: CI tables held 35.69% of the data and carried 48.98% of writes per second — against merge requests at 20% of writes and a long tail of everything else. Splitting the database in half by write traffic would be the optimal scaling step, and CI was that half. Instinct had pointed the same way (ci_-prefixed tables were already among the largest), and the measurement confirmed it; in the end only three CI tables lacked the prefix.
Solution
An August 2021 proof of concept established viability by brute honesty: separate the databases, see what breaks, fix or file until the application 'pretty much works' — never merged, progressively decanted into small merge requests routed to owning teams. What the PoC couldn't solve was time: the application changes took nearly a year, against a codebase evolving under many engineers who had never heard of the decomposition. The answer was a ratchet, inspired by how GitLab handles new RuboCop rules. Detection first: a cross-join analyzer erroring on any query spanning both databases; a cross-database transaction analyzer erroring when one transaction touched both; query-analyzer metrics sampled at 1 in 10,000 (parsing is expensive) into Prometheus, confirming the burn-down was real in production and catching code paths no test covered; a RuboCop rule banning bare ActiveRecord::Base so every model declared its database. Then the allowlist: existing violations enumerated as exceptions, and any new violation fails the pipeline. The moving target stopped moving — a clear list, visible progress, and no way to regress.
Two mechanisms replaced what the split destroyed. Foreign keys between CI and non-CI tables became loose foreign keys: Postgres on-delete triggers — which, unlike ActiveRecord callbacks, cannot be skipped and catch bulk deletes — write the parent deletion into a queue table, and a periodic Sidekiq worker cleans up child records asynchronously, with cascade semantics (DELETE or NULLIFY) preserved. The async form paid an unexpected dividend: enormous cascading deletes that used to cause timeouts now proceed in controlled batches that never overload the database. Where joins couldn't be refactored away at all — CI runners' deep dependence on projects and namespaces — the relevant columns are mirrored into the CI database, guarded by periodic consistency checking that compares batches of mirrored rows against expected values, schedules fixes for discrepancies, and advances a Redis cursor. The honest accounting: mirroring was expected to be needed in a few places and ended up needed in exactly two tables.
The rollout converted a discrete change into a continuous one. The insight: reconfigure the application to behave as though it has two databases — two connection sets, separate write proxies, CI reads served from a dedicated standby cluster — while everything still points at one physical database. That made phases 1 through 6 individually trivial to roll back and safe to ship early, and left Phase 7, the real migration, as 'trivial reconfiguration of a single database host' — 193 labeled issues across seven phases, distributed across teams by scoped labels with per-phase deadlines. For the cutover itself, the team designed the near-zero-downtime version — Patroni cascading replication to the CI cluster, pause CI writes at PGBouncer, capture the primary's LSN, wait for catch-up, promote, repoint — and declined it, on three stated grounds: no easy rollback (CI writes during the migration would be lost), a few-seconds error window that muddies the success signal, and no hard business requirement. The chosen plan took a two-hour window with traffic blocked at the CDN, a three-step rollback (repoint reads, repoint writes, bump sequences past test data), and seven full staging rehearsals that surfaced many small issues and let every participant perfect their steps. Production took 93 minutes. The results held: primary CPU headroom recovered, 9.2TiB of the Main database and 12.5TiB of the CI database freeable by truncating the other side's tables, dead tuples down, vacuum saturation from 80–100% peaks to a stable ~15% on both databases, and Sidekiq's average query duration down at least five-fold once connection limits — previously throttled to protect the primary — could be raised into the new headroom.
Tradeoffs
- Replicas scale reads; the primary is the ceiling. Patroni pools and PGBouncer bought years and could never touch the bottleneck, because every write converges on one host and the host tops out — 96 vCPUs and no larger machine. The class's purest lesson lives in that asymmetry: read scaling is a solved, incremental problem; write scaling on a single cluster is a wall, and it arrives while your read dashboards still look healthy.
- Split where the measurement says, and let instinct be confirmed rather than trusted. The write-traffic analysis made CI the optimal halving — ~49% of writes, ~36% of data — and the ci_ prefix made it convenient; only three unprefixed tables spoiled the elegance. Decomposition's value is bounded by the coupling of what you extract: the deciding table (reads/s, writes/s, size, per table group) is one afternoon of analysis that prevents a year of extracting the wrong half.
- A moving target requires a ratchet, not a sweep. A year of changes against a codebase evolving under engineers who'd never heard of the project would have accumulated new violations faster than heroes could remove old ones — so detection (cross-join and cross-transaction analyzers, 1-in-10,000 sampled query metrics to Prometheus), an allowlist of existing offenders, and pipeline failure on anything new converted the chase into a monotonic burn-down. The production sampling earned its keep twice: proving the burn-down real, and catching paths no test executed.
- Cross-database referential integrity is rebuilt as asynchronous convergence, and the rebuild fixed an old wound. Loose foreign keys — on-delete triggers (unskippable, bulk-safe, unlike model callbacks) feeding a queue drained by Sidekiq — preserve cascade semantics without cross-DB constraints, at the price of convergence windows and a queue to operate. The dividend: giant cascading deletes that once caused timeouts now proceed in controlled batches. Where refactoring couldn't kill a join, two tables' columns are mirrored with periodic consistency checking — expected in a few places, needed in exactly two.
- Make the discrete change continuous: run the application as if it has two databases while both connections point at one. That single insight decomposed an atomic 1→2 migration into seven phases where 1–6 ship early, carry trivial rollback, and introduce almost no risk — leaving the real cutover as reconfiguration of one host. 193 issues under scoped phase labels distributed the work across an async, all-remote organization without a war room; the phase label WAS the deadline.
- Zero downtime was designed, evaluated, and declined on purpose. The near-zero plan (pause CI writes, capture LSN, await catch-up, promote) lost to a two-hour window on three stated grounds: rollback would lose CI writes written mid-migration, a few-seconds error window muddies the did-it-work signal, and no business requirement demanded zero. Seven staging rehearsals priced the runbook and its three-step rollback; production ran in 93 minutes. Downtime is a cost — rollback clarity is a benefit — and the team did the arithmetic out loud instead of worshipping zero.
Patterns in this article
- Loose Foreign Keys
Minted under GitLab's own coinage, which describes the mechanism exactly: replace cross-database foreign keys with Postgres on-delete triggers — unskippable and bulk-delete-safe, guarantees ActiveRecord callbacks cannot give — writing parent deletions to a queue table that a periodic worker drains, preserving cascade DELETE/NULLIFY semantics as asynchronous convergence. The in-source dividend defines the pattern's second face: cascades in controlled batches also cured the timeout pain of giant synchronous deletes.
- Violation Ratchet
Minted from the moving-target section, inspired by GitLab's RuboCop-exception workflow: implement detection for the invariant being established (cross-join and cross-database-transaction analyzers, sampled production query metrics), allowlist every existing violation, and fail the pipeline on any new one — converting an unwinnable chase against a codebase evolving under uninvolved engineers into a monotonic burn-down with visible progress. CATEGORY-STRAIN FLAG: this is an engineering-process pattern enforced by CI rather than a runtime systems pattern; filed under consistency (it defends an architectural invariant during transition) pending owner ruling.
- Universal Staged Rollout
Fourth consecutive datastore migration carrying the pattern, and this instance contributes its sharpest move: the as-if reconfiguration — run the application with two full connection topologies while both point at one physical database — which converts an atomic 1→2 migration into seven phases, six of them trivially rollbackable and shippable early. Plus the rehearsal discipline priced in numbers: seven full staging runs against a two-hour budget, production done in 93 minutes.
Also solving this
Other systems in behindscale's Single-cluster scaling ceiling class: