Half the Writes Must Go: GitLab's Database Decomposition
GitLab.com ran on one big Postgres database holding almost all user data (git data aside). For years they scaled it by adding read-only copies and pooling database connections. That bought room but never touched the real limit: every write still had to go to one main server, and that server was already the largest machine available. The long-term fix was to spread the data across many servers by customer account, but the app had never kept each account's data cleanly separate, so that was a huge job. So the team took a reachable step first: split the database in two. Measurement chose where to cut: the CI tables carried about half the writes, so moving them out roughly halved the load. Preparing the split took about a year. The final switch-over was kept small by making the app talk to two databases well before the data was actually split. They designed a near-zero-downtime switch, then chose a planned two-hour outage instead, because it was far easier to undo. After seven full practice runs, the real switch took 93 minutes.
Add read-only copies to a write-heavy database and see why they do not help; ask for a bigger machine than 96 cores and find none. Run the write analysis, turn on the gate that blocks new cross-database queries and watch it stop a fresh one, and replace the foreign keys with queues. Then make the app run as if it had two databases on one host, rehearse seven times, read why the team turned down the zero-downtime plan, and cut over in 93 minutes.
Problem
Before the split, GitLab already ran a well-scaled single database: one Postgres holding almost all the data GitLab.com users create, git data aside. In front of it sat two standard scaling tools. Patroni managed a pool of read-only copies of the database, called replicas, so read queries could be spread across many machines. PGBouncer pooled connections, letting a huge number of app servers share a small, safe number of database connections. Both help enormously with reads, but neither touches the real bottleneck, because every write still has to go to the one main server. That server was a single machine with 96 CPU cores, already close to the largest a single machine can be. Scaling it up further would eventually hit a wall, and even an impossibly large machine would still leave a 22TiB database that keeps getting harder to run and had become a frequent source of outages.
The database team, formed in early 2021, first aimed at the long-term answer: spread the data across many servers by customer account (in GitLab terms, the top-level namespace that owns projects). That ran into the application's past. GitLab had never been built to keep each account's data cleanly separate, so this turned out to be very hard. The team's judgment was careful: this kind of split is probably where the design ends up, but they needed something sooner. So they picked a reachable move and kept the bigger one on the map. The reachable move was to pull one group of tables out into a second database. That only works if you can find a group that is not tightly linked to the rest, because every query that reaches across the new boundary has to be removed first.
The place to cut came from measurement, not instinct. Looking at where data sat and where writes landed gave a clear answer: the CI tables held about 36% of the data but took 49% of the writes each second. Merge-request tables were a distant second at 20% of writes, and everything else trailed off from there. Cutting the database in half by write traffic was the best single scaling step, and CI was that half. Instinct had pointed the same way, since the CI tables (mostly named with a ci_ prefix) were already among the biggest, and the numbers confirmed it. Only three CI tables turned out to be missing that prefix.
Solution
In August 2021 the team built a proof of concept the blunt way: split the databases, see what broke, and fix or note each thing until the app ran. They never merged it; they broke the changes into small merge requests handed to the teams that owned the code. The hard part was time. The changes took nearly a year, while hundreds of engineers who had never heard of the project kept changing the same code, so new problems appeared faster than old ones were fixed. GitLab's answer was borrowed from how it rolls out new code-style rules with RuboCop, a checker for Ruby code: detect the problem automatically, list every existing case, then block any new case from merging. Detection meant automatic checks. A query or transaction touching both databases now fails, every model must name its database instead of using Rails' generic default, and a 1-in-10,000 sample of live queries went to Prometheus to confirm the count was really falling. Then the allowlist: every existing violation became a known exception, and any new one failed the pipeline. Now the list could only shrink.
Splitting the database broke two things that needed rebuilding. First, some foreign keys linked CI tables to the rest. A foreign key is the rule that when a parent row is deleted, its child rows are cleaned up too. You cannot keep a foreign key across two separate databases, so GitLab replaced them with what it calls loose foreign keys. A Postgres trigger fires whenever a parent row is deleted (triggers cannot be skipped and also catch bulk deletes, unlike Rails' own delete code) and writes that deletion into a queue table. A background job then reads the queue and cleans up the child rows a moment later, still honoring delete or set-to-null. In the background, huge chained deletes that used to time out now run in small, steady batches. Second, a few queries could not be untangled: CI runners lean heavily on the projects and namespaces tables. For those, GitLab copies the needed columns into the CI database. A periodic check keeps the copy honest: it compares batches of copied rows against the originals, fixes any that drifted, and remembers its place with a marker in Redis. They expected to need this in a few places; it turned out to be just two tables.
The switch-over turned one big, risky change into many small safe ones. The key idea: make the app behave as if it already had two databases (two sets of connections, separate write paths, CI reads from their own standby copy) while everything still pointed at the one real database. GitLab did this across seven phases. The early phases stood up the second database's machines and split first the reads, then the write paths, all still landing on the original server, so each step was easy to undo and safe to ship early. That left only the final phase, the real migration, as a tiny change: point one database host at the new server. The work was tracked as 193 tickets across the seven phases, handed to teams by labels, each with a rough deadline.
For the cutover, the team first designed a near-zero-downtime version. It would copy everything to the CI cluster, briefly pause CI writes, note how far the main database had gotten, wait for the copy to catch up, then promote it and send writes there. They designed it fully, then chose not to use it, for three plain reasons. If something went wrong they could not cleanly undo it, since any CI writes made mid-switch would be lost. The few-second window of errors would blur the signal of whether the switch had worked. And nothing about the business actually required zero downtime. The plan they chose took a two-hour outage, with user traffic blocked at the CDN, and a simple three-step undo: point reads back, point writes back, then nudge the ID counters past any test data. Seven full practice runs on staging surfaced many small issues and let everyone rehearse. The real run took 93 minutes.
The results held up. The main server got its CPU headroom back. Dead rows, the old versions of rows that Postgres has to clean up, dropped sharply, and that cleanup, which had been running flat out at 80-100%, settled to a steady 15% on both databases. Each database could also free space (about 9.2TiB on one and 12.5TiB on the other, out of 22TiB) by dropping the tables that now lived on the other side. And once the team could safely raise connection limits into the new headroom, the average query time for background jobs dropped at least five-fold.
Tradeoffs
- Read-only copies scale reads; the one main server is the ceiling for writes. Adding copies and pooling connections bought years but never touched the bottleneck, because every write still lands on that one server, and it was already the biggest machine available. That gap is the whole lesson: scaling reads is a steady, solved problem, while scaling writes on a single database is a wall, and it arrives while your read dashboards still look healthy.
- Cut where the measurement points, and let instinct only confirm it. The write-traffic numbers made CI the obvious half to move out (about 49% of writes, 36% of the data), and its ci_ naming made it convenient, with only three CI tables missing the prefix. How much a split like this helps depends on how loosely the piece you pull out is connected to the rest. Measuring reads, writes, and size per group of tables is an afternoon of work that saves a year spent pulling out the wrong half.
- You cannot fix a moving target by hand; you need a gate that blocks new problems while you clear the old ones. Over a year, engineers who had never heard of the project kept adding new violations faster than they could be removed. So the team added automatic checks that catch any query or transaction spanning both databases, recorded every existing violation as a known exception, and made any new one fail the build. After that, the count could only fall. The live sampling earned its keep twice: it proved the count was really dropping, and it caught code paths no test ran.
- When you cannot keep foreign keys across two databases, you rebuild the same guarantee a slower way. GitLab's loose foreign keys use a Postgres trigger that fires on every delete (even bulk deletes, which some app-level code misses) to queue the parent deletion. A background job then cleans up the child rows a moment later, still honoring delete or set-to-null, at the cost of a short delay and a queue to run. Rebuilding it this way also fixed an old problem: the giant chained deletes that used to time out now run in small batches. Where a query could not be untangled at all, the needed columns are copied into the other database and kept in sync by a periodic check, needed in the end for just two tables.
- Turn one big change into a series of small ones: make the app behave as if it already had two databases while both connections still point at one. That one idea split a single hard migration into seven phases. The first six could ship early, were easy to undo, and added almost no risk, leaving the real switch-over as a tiny change to one host. The work went out as 193 tickets tagged by phase, spread across a remote team with no all-hands war room. Because each team knew when its phase was due, the tag itself set the deadline.
- Zero downtime was designed, weighed, and turned down on purpose. The near-zero plan (pause CI writes, note the log position, wait for the copy to catch up, promote) lost to a planned two-hour outage for three reasons. Rolling back would lose any CI writes made mid-switch. The few-second window of errors would blur the signal of whether it worked. And no business need demanded zero downtime. Seven staging rehearsals tested the plan and its three-step undo, and the real run took 93 minutes. Downtime is a cost and easy rollback is a benefit; the team weighed the two openly instead of treating zero downtime as the default goal.
Patterns in this article
- Loose Foreign Keys
GitLab's own name for this, and it fits: replace the foreign keys between the two databases with Postgres triggers. A foreign key normally makes the database clean up child rows when their parent is deleted. Since that cannot work across two databases, a trigger fires on each delete (including bulk deletes, which some app-level code misses) and adds the parent to a queue. A background job then cleans up the child rows a moment later, still honoring delete or set-to-null. A nice side effect: the giant chained deletes that used to time out now run in small, steady batches.
- Violation Ratchet
GitLab borrowed this from how it rolls out new code-style rules. First, add automatic checks that catch the problem you are stamping out; here, any query or transaction that touches both databases. Next, record every case that already exists as a known exception. Then make any new case fail the build, so the count can only go down, never up. That one-way behavior is what GitLab named a ratchet.
- Universal Staged Rollout
The rollout's sharpest move: make the app behave as if it already had two databases (two full sets of connections) while both still point at one real database. That turns a single hard migration into seven phases, the first six easy to undo and safe to ship early, leaving the real switch as a tiny change to one host. Seven full practice runs against a two-hour budget brought the real run in at 93 minutes.
Also solving this
Other systems in behindscale's Single-cluster scaling ceiling class: