Virtual Before Physical: Partitioning GitHub's Relational Databases
For over a decade, GitHub.com revolved around one MySQL cluster — mysql1 — holding users, repositories, issues, and pull requests. Between 2019 and 2021, GitHub partitioned it without downtime by doing the work in two strictly ordered stages: first virtually, with schema domains and SQL linters that made cross-domain queries a failing test before any data moved, and only then physically, with Vitess vertical sharding and a custom write-cutover whose read-only window lasted tens of milliseconds. They moved 130 of their busiest tables in one cutover. The result: the same data now answers 1.2M queries/s across several clusters while average per-host load halved.
Lint the gists domain down to zero violations to unlock the move — then run the six-step write-cutover and watch the read-only window close in tens of milliseconds.
Problem
More than ten years ago, GitHub.com started the way many web applications of its era did: Ruby on Rails in front of a single MySQL database holding most of its data. The architecture iterated as the company grew — data for some features like statuses moved to separate MySQL databases, read replicas spread query load across machines, and ProxySQL reduced the number of connections opened against primaries. But at its core, GitHub.com remained built around one main cluster, called mysql1, housing a large portion of the data behind core features: user profiles, repositories, issues, pull requests.
Growth turned that center of gravity into a double bind. On capacity, GitHub struggled to keep the cluster adequately sized, perpetually moving to newer and bigger machines to scale up — a strategy with a visible end. On availability, mysql1 was a shared-fate domain: any incident that touched it degraded every feature storing data there, which was most of the features that make GitHub GitHub.
The deeper obstacle was in the application, not the database. A decade of code had been written against the guarantee that every table lived in one MySQL database — JOINs freely crossed feature boundaries, and MySQL transactions provided consistency across any set of tables. The moment tables move to separate clusters, both guarantees silently break: a cross-cluster JOIN is impossible, and a transaction spanning two databases can no longer guarantee consistency. Before any data could move, GitHub had to find every one of those hidden dependencies — and stop new ones from being written faster than old ones were removed. In 2019, they set a plan in motion to build exactly that tooling. In 2019, mysql1's tables answered 950,000 queries per second on average.
Solution
The plan's defining property is its ordering: tables were partitioned virtually, in the application layer, before anything moved physically. The unit of virtual partitioning is the schema domain — a declared set of tightly coupled tables frequently used together in queries and transactions, codified in a YAML file in the Rails app (the gists domain holds gists, gist_comments, and starred_gists; repositories holds issues, pull_requests, repositories). A linter keeps the file in lockstep with the actual schema and forces every table to belong to a domain: the boundaries exist in code, reviewed like code.
Two SQL linters then give those boundaries teeth. The query linter raises an exception in development, test, and CI whenever a single query references tables from different domains, so developers hit violations at their desks rather than in production, and no new coupling lands by accident. Existing violations are annotated with a special comment — cross-schema-domain-query-exempted — which both suppresses the error and builds a visible backlog of queries that must be fixed before a move. GitHub upstreamed tooling to Rails to help burn that backlog down: an annotate method on ActiveRecord relations, and a disable_joins option for has_many :through associations that replaces cross-domain JOINs with sequential queries on primary keys. Other exemptions fall to preload instead of includes, or to joining in the application — two queries whose union happens in Ruby, which occasionally outperforms MySQL's own planner on unstable query plans. Every such change ships behind a Scientist experiment running old and new implementations side by side on live traffic.
Transactions get their own linter, because they break differently: MySQL guarantees consistency across tables only within one database, so a transaction spanning future partition boundaries loses its guarantee silently. The transaction linter runs in production under heavy sampling, mapping where cross-domain transactions actually happen. Where transactional consistency is essential, the data model changes instead of the code: polymorphic tables that mixed domains — a reactions table storing rows for issues, pull requests, and discussions alike — are extracted into per-domain tables so the rows that must commit together stay on the same cluster. A domain with no violations is virtually partitioned: the application already behaves as if the split had happened, while every byte still sits on mysql1.
Only then does data move, and GitHub built two independent ways to move it. Vitess — the MySQL scaling layer originally built at YouTube — provides vertical sharding: VTGate proxies deployed in Kubernetes speak the MySQL protocol to the application while VReplication streams tables between clusters behind the scenes. But because Vitess adoption was still early at GitHub in 2020, they also built a deliberately boring alternative for moving large sets of tables at once: the write-cutover process, riding on stock MySQL replication and ProxySQL. The destination cluster is first attached as a replica of the source; ProxySQL multiplexes client connections so routing can change in one place. Then a script executes six steps: enable read-only on the source primary, read its last executed GTID, poll the destination until that GTID arrives, stop replication, flip ProxySQL routing to the destination primary, and disable read-only on both. After thorough preparation and rehearsal, those steps execute in tens of milliseconds for GitHub's busiest tables — run in the lowest-traffic window, they produce only a handful of user-facing failed writes. This is the process that split mysql1 itself: 130 of the busiest tables, the ones powering repositories, issues, and pull requests, moved in one cutover.
By 2021, the same tables answered 1,200,000 queries per second across several clusters — up more than 30% from 2019 — while the average load on each host halved, a 50% reduction that the post credits with a significant drop in database-related incidents. Horizontal sharding — splitting individual tables across clusters — was explicitly deferred to a future post: this one is about buying headroom by moving whole domains.
Tradeoffs
- Exemption annotations are institutionalized debt, and the linter alone never pays it down. The query linter's suppression comment converts each existing violation into a tracked backlog item — which is honest bookkeeping, but the backlog must reach zero for a domain before its tables can move, and nothing about the linter makes that happen. The mechanism stops new coupling from landing; retiring old coupling is sustained human work across every team whose feature touches the domain. A migration gated on linters is really gated on an organization's willingness to burn down a backlog it can technically ignore.
- Replacing database JOINs with application-side joins trades one set of guarantees for another. Two sequential queries whose results are unioned in Ruby are not the same operation as one JOIN: the reads see the database at two different moments, so the atomicity of a single consistent snapshot is quietly gone, and the rows fetched by primary key can shift between the queries. GitHub found the trade sometimes runs in their favor — MySQL's planner produces unstable plans on certain shapes, and the application-side version has a flatter performance profile — but the consistency cost is real and permanent, which is exactly why each change shipped behind a Scientist experiment rather than on faith.
- Preserving transactional consistency by reshaping the data model couples the schema to deployment topology. Extracting polymorphic tables — splitting one reactions table into per-domain tables so co-committing rows stay on one cluster — is the correct move when the transaction matters, but it means the logical data model now bends to where data physically lives. Denormalization by partition boundary is a new kind of coupling introduced to remove an old one, and every future feature that wants to span domains inherits the constraint.
- Running two migration mechanisms is a deliberate redundancy with a real carrying cost. The write-cutover exists because betting GitHub.com's availability on early-stage Vitess adoption was an unacceptable single point of failure in the plan itself — and factors like deployment topology and read-your-writes support kept Vitess from being the right tool for every move. The judgment call favors risk mitigation over elegance: two tools, two operational surfaces, two sets of failure modes to rehearse, in exchange for never being stuck when one of them doesn't fit.
- The cutover buys certainty with a small, deliberate outage. For tens of milliseconds, the busiest tables at GitHub accept no writes, and the web requests that try get 500s — a handful of real users see real errors, by design, in the lowest-traffic window. The alternative styles (dual-write reconciliation, long read-only windows) spread smaller risks over longer periods; GitHub chose a brief, certain, rehearsed failure whose blast radius is measured before it happens. The GTID poll in step three is the load-bearing detail: the window stays short only if the destination is already caught up, which is why the process demands the thorough preparation and exercising the post mentions.
- Vertical partitioning moves whole domains — so a single domain remains the next ceiling. Moving the repositories domain off mysql1 buys headroom, but issues, pull_requests, and repositories still live together and still grow together; nothing in this post's tooling splits one huge table across clusters. GitHub says so directly, deferring horizontal sharding to a future post. This is precisely where Figma's story begins — when the unit that outgrows a machine is no longer the cluster but the table, vertical partitioning has nothing left to move, and the sharding problem starts in earnest.
Patterns in this article
- Logical–Physical Migration Split
GitHub's entire plan is the split's first stage made rigorous: schema domains and SQL linters make the application behave as if partitioned — with violations failing CI — while every byte still sits on mysql1. Only a domain with zero violations earns a physical move. Where Figma gated the physical stage on a feature-flag ramp, GitHub gates it on a linter reaching zero: the same pattern, enforced by the build instead of by traffic.
- Fault Isolation
The availability half of GitHub's motivation is blast radius: any incident on mysql1 degraded every core feature at once. Partitioning by schema domain shrinks the shared-fate domain — an incident on one cluster now touches one feature family — which the post credits, alongside the load reduction, for the significant drop in database-related incidents.
- Compile-Time Boundary Enforcement
GitHub's query linter turns an architectural intention — these tables will live apart — into a failing test at the developer's desk, years before the tables actually move. The transaction linter extends the same enforcement to production under sampling, catching what static analysis of queries can't see. The boundary is real from the day it's declared, not the day it's deployed.