Virtual Before Physical: Partitioning GitHub's Relational Databases
For over a decade, GitHub.com revolved around one MySQL cluster, mysql1, holding users, repositories, issues, and pull requests. Between 2019 and 2021, GitHub partitioned it without downtime, and the order was the point: first in the code, then in the database. In the code, they grouped related tables into named schema domains and added linters that made any query crossing a domain boundary fail the build, so the app behaved as if split before a single byte moved. Only then came the database move: whole tables shifted to new clusters with Vitess (the MySQL scaling layer born at YouTube) and a custom write-cutover whose read-only window lasted tens of milliseconds. They moved 130 of their busiest tables in one cutover. The result: the same data now answers 1.2M queries/s across several clusters, up from 950k, while average per-host load halved.
Lint the gists domain down to zero violations to unlock the move, then run the six-step write-cutover and watch the read-only window close in tens of milliseconds.
Problem
More than ten years ago, GitHub.com started the way many web applications of its era did: Ruby on Rails in front of a single MySQL database holding most of its data. The architecture iterated as the company grew: data for some features like statuses moved to separate MySQL databases, and read replicas spread query load across machines. A connection pooler called ProxySQL cut the number of open connections to the primary databases. But at its core, GitHub.com remained built around one main cluster, called mysql1, housing a large portion of the data behind core features: user profiles, repositories, issues, pull requests.
Growth made that single center of gravity a problem on two fronts at once. On capacity, GitHub struggled to keep the cluster adequately sized, perpetually moving to newer and bigger machines to scale up, a strategy with a visible end. On availability, everything shared mysql1's fate: any incident that touched it degraded every feature storing data there, which was most of what makes GitHub GitHub.
The deeper obstacle was in the application, not the database. A decade of code had been written against the guarantee that every table lived in one MySQL database: JOINs freely crossed feature boundaries, and MySQL transactions provided consistency across any set of tables. The moment tables move to separate clusters, both guarantees silently break: a cross-cluster JOIN is impossible, and a transaction spanning two databases can no longer guarantee consistency.
Before any data could move, GitHub had to find every one of those hidden dependencies, and stop new ones from being written faster than old ones were removed. In 2019, they set a plan in motion to build exactly that tooling. That year, mysql1's tables answered 950,000 queries per second on average.
Solution
The plan's defining property is its ordering: tables were partitioned virtually, in the code, before anything moved physically. The unit of virtual partitioning is the schema domain, which is simply a named group of tables that belong together because they are used in the same queries and transactions. The groups are written down in a YAML file in the Rails app: the gists domain holds gists, gist_comments, and starred_gists, and the repositories domain holds issues, pull_requests, and repositories. A linter keeps that file in step with the real database and forces every table to belong to a domain, so the boundaries live in code and get reviewed like code.
Two SQL linters then enforce those boundaries. The query linter raises an error in development, test, and CI whenever one query touches tables from different domains, so developers hit it at their desks, and no new coupling slips in by accident. Existing violations are marked with a special comment, cross-schema-domain-query-exempted, which silences the error for now but adds the query to a backlog that must be cleared before a move. GitHub added tools to Rails to help clear that backlog. One attaches the exemption comment automatically. Another lets a has_many :through association (a Rails way of reaching related rows through a link table) run as separate lookups by primary key instead of a cross-domain JOIN. Other queries are fixed by loading the two tables in separate steps, or by joining in the application: run two queries and combine the results in Ruby, which can even beat MySQL's own planner on unstable query plans. Every such change ships behind a Scientist experiment: old and new versions both run on the same real requests and their results are compared, but only the old result is used, so users are never affected.
Transactions get their own linter, because they break differently: MySQL keeps a transaction consistent only when all its tables live in one database, so a transaction that would span two future databases loses that guarantee silently. The transaction linter runs in production on a small sample of traffic, mapping where cross-domain transactions actually happen. Where a transaction's consistency really matters, GitHub changes the data model instead of the code. A polymorphic table is one shared table that stores rows for several features at once, such as a single reactions table holding reactions for issues, pull requests, and discussions. Those are split into one table per domain, so the rows that must commit together stay on the same cluster. A domain with no violations left is virtually partitioned: the app behaves as if split, while every byte still sits on mysql1.
Only then does data actually move, and GitHub built two independent ways to do it. The first is Vitess, the MySQL scaling layer born at YouTube: it puts a proxy in front that speaks the normal MySQL protocol, so the app does not notice, and copies whole tables between clusters in the background. Because Vitess was still new at GitHub in 2020, they also built a deliberately boring second method for moving many tables at once: a write-cutover that rides on ordinary MySQL replication and ProxySQL. First the new cluster is set up as a live copy of the old one; ProxySQL sits in front and shares the app's database connections, so all traffic can be redirected from one place.
Then a script runs six quick steps:
- Turn on read-only on the old primary, so it refuses all writes.
- Read the old primary's last write position (its GTID, a marker of exactly how far its writes have gotten).
- Wait until the new cluster has caught up to that same position.
- Stop the new cluster copying from the old one.
- Point ProxySQL at the new primary.
- Turn read-only back off, so writes resume on the new cluster.
Those six steps run in tens of milliseconds for the busiest tables, and at the low-traffic hour only a handful of writes fail. This is how mysql1 itself was split: 130 of the busiest tables, powering repositories, issues, and pull requests, moved in one cutover.
By 2021, the same tables answered 1,200,000 queries per second across several clusters, up from 950,000 two years earlier, while the average load on each host halved. The post credits that 50% load reduction with a significant drop in database-related incidents. Horizontal sharding, which means splitting one big table across several clusters, was left for a future post; this one is about buying headroom by moving whole domains.
Tradeoffs
- An exemption annotation is a comment that tells the linter to ignore a query that still crosses a domain boundary. Each one is honest bookkeeping: it records a violation that has to be removed later. But the linter only stops new violations from landing; it never removes the old ones. Clearing them is steady human work across every team whose feature touches the domain, and the backlog must reach zero before a domain's tables can move. So a migration that looks gated on linters is really gated on whether the organization will clear a backlog it could otherwise ignore forever.
- Replacing database JOINs with application-side joins trades one set of guarantees for another. Two sequential queries whose results are unioned in Ruby are not the same operation as one JOIN. The reads see the database at two different moments, so the atomicity of a single consistent snapshot is quietly gone, and the rows fetched by primary key can shift between the queries. GitHub found the trade sometimes runs in their favor: MySQL's planner produces unstable plans on certain shapes, and the application-side version has a flatter performance profile. But the consistency cost is real and permanent, which is exactly why each change shipped behind a Scientist experiment rather than on faith.
- Preserving consistency by reshaping the data model ties the schema to where the data physically lives. Splitting one shared reactions table into per-domain tables, so rows that must commit together land on the same cluster, is the right move when the transaction matters. But now the logical data model bends to the physical layout: the schema depends on which tables sit on which cluster (its deployment topology). This is a new kind of coupling introduced to remove an old one, and every future feature that wants to span domains inherits the constraint.
- Running two migration mechanisms is a deliberate redundancy with a real carrying cost. The write-cutover exists because betting GitHub.com's availability on early-stage Vitess adoption was an unacceptable single point of failure in the plan itself. And factors like deployment topology and read-your-writes support (being able to read data you just wrote) kept Vitess from being the right tool for every move. The judgment call favors risk mitigation over elegance: two tools, two operational surfaces, two sets of failure modes to rehearse, in exchange for never being stuck when one of them doesn't fit.
- The cutover buys certainty with a small, deliberate outage. For tens of milliseconds the busiest tables accept no writes, and the web requests that try get 500s: a handful of real users see real errors, by design, in the lowest-traffic window. Two gentler-looking alternatives each spread smaller risks over a longer time. Dual-write reconciliation writes to both the old and new database for a while and keeps checking that they agree, which avoids downtime but adds a long, complex period where the two can drift. A long read-only window keeps the tables readable but blocks writes for minutes, trading one short outage for a long one. GitHub instead chose a brief, certain, rehearsed failure whose blast radius is measured beforehand. The load-bearing step is the wait in step three, when the script pauses until the new cluster has caught up to the old one's last write position. The window stays short only if that catch-up is already almost done, which is why the process demands so much preparation and rehearsal.
- Vertical partitioning moves whole domains, so a single domain remains the next ceiling. Moving the repositories domain off mysql1 buys headroom, but issues, pull_requests, and repositories still live together and still grow together; nothing in this post's tooling splits one huge table across clusters. GitHub says so directly, deferring horizontal sharding to a future post. This is precisely where Figma's story begins: when the unit that outgrows a machine is no longer the cluster but the table, vertical partitioning has nothing left to move, and the sharding problem starts in earnest.
Patterns in this article
- Logical–Physical Migration Split
GitHub's entire plan is the split's first stage made rigorous: schema domains and SQL linters make the application behave as if partitioned (with violations failing CI) while every byte still sits on mysql1. Only a domain with zero violations earns a physical move. Where Figma gated the physical stage on a feature-flag ramp, GitHub gates it on a linter reaching zero: the same pattern, enforced by the build instead of by traffic.
- Fault Isolation
The availability half of GitHub's motivation is blast radius: any incident on mysql1 degraded every core feature at once. Partitioning by schema domain shrinks the shared-fate domain (an incident on one cluster now touches one feature family), which the post credits, alongside the load reduction, for the significant drop in database-related incidents.
- Compile-Time Boundary Enforcement
GitHub's query linter turns an architectural intention (these tables will live apart) into a failing test at the developer's desk, years before the tables actually move. The transaction linter extends the same enforcement to production under sampling, catching what static analysis of queries can't see. The boundary is real from the day it's declared, not the day it's deployed.
Also solving this
Other systems in behindscale's Single-cluster scaling ceiling class: