The Largest Shard Money Could Buy: Slack's Road to Vitess

The original architecture is laid out with unusual completeness: three families of MySQL clusters.

• Shards held virtually all customer data (messages, channels, DMs), split by workspace id, thousands of workspaces per shard, all of a workspace's data on one shard so the application talked to exactly one database.
• A metadata cluster mapped each workspace id to its shard id, and was checked first on every access.
• A kitchen-sink cluster held data tied to no workspace, like the app directory.

THE ORIGINAL LAYOUT

Every shard was at least two MySQL instances in different data centers, active-active (both sides take reads and writes, copying to each other in the background), a hash choosing the preferred side and the other absorbing retries on failure. The webapp, Slack's single big application, owned all of it: the metadata lookup, the connection routing, the sharding logic.

The post is fair to this design before it buries it: high availability from the dual-write pair, fast product work from the everything-in-one-place model, minutes-to-diagnosis debugging, and easy growth (more teams, more shards). Then the bolded question: what if a single team and all of their Slack data doesn't fit our largest shard?

By fall 2016, with hundreds of thousands of MySQL queries per second across thousands of sharded hosts, the disadvantages had names.

• Scale limits: the largest customers' shards had reached the largest available hardware, regularly maxing out what a single host could take.
• One data model: Enterprise Grid (which spans many workspaces) and what became Slack Connect (which shares channels across companies) broke the assumption that a team's data lives on one shard, adding complexity and, in cases, a speed penalty.
• Hot spots beside waste: unable to spread a big customer's load, Slack ran a few burning shards next to a mostly idle long tail, overprovisioned because splitting shards and moving teams was hard and usage was unpredictable.
• Availability coupling: every core feature needed the team's shard up, so one shard's outage was a full Slack outage for every customer on it.
• Operations: the active-active setup was nonstandard MySQL that needed a lot of custom tooling, and with no replicas in the layout and the app routing straight to hosts, read replicas could not be used safely at all.

TWO AXES, ONE WALL

These are the five walls the migration set out to remove.

The choice was evolve in place or replace. The in-place option was to teach the webapp's existing routing layer to use a finer shard key like channel id (one Slack channel) instead of the whole workspace. It was prototyped seriously and rejected on the evidence: the application logic and the storage layout were deeply tangled (fetching a channel's message count assumed which team's shard held it; multi-workspace organizations were handled by checking several shards by hand). Untangling would be slow, and the approach fixed neither operations nor replica use, with the post's long-term verdict being that a surprisingly hot single-team shard still would not be easy to scale. One constraint shaped everything else: stay on MySQL. Thousands of distinct queries, some using MySQL-only features, plus years of operational practice (deployment, durability, backups, data-warehouse export, compliance), ruled out the alternatives. Non-relational stores like DynamoDB or Cassandra and newer distributed-SQL systems like Spanner or CockroachDB would all have been far more disruptive changes for a deliberately conservative infrastructure culture.

Vitess fit the requirements as built:

• MySQL Core: MySQL stays the real storage and replication engine underneath.
• Sharding: the sharding lives in the database layer, flexible and growable without adding application logic.
• Operability: automatic primary failovers and backups come built in, and a lock server tracks the layout (the topology) so the application can ignore where anything lives.
• Extensibility: it is open-source Go that Slack could extend, which it did, becoming one of the project's biggest contributors (topology metadata refactored for isolation regions, MySQL gaps closed, migration and load-testing tools, and Prometheus, Orchestrator, and xtrabackup integrations).

The migration was staged with the care of a system that cannot stop running. The first end-to-end use case in production was deliberately small, RSS feeds into channels, chosen because it forced the real work into existence: provisioning, service discovery, backup and restore, topology management (tracking which servers hold what), and credentials. Then a generic backfill system cloned existing tables while the application wrote to both old and new (double-writes), and a parallel system read from both and compared the results (double-read diffing), proving the Vitess-backed tables behaved identically before any cutover. Three years later: 99% of all MySQL traffic on Vitess, with the last 1% scheduled within two months. At peak it served 2.3 million queries per second (2 million reads, 300 thousand writes), with 2ms typical latency and 11ms for the slowest 1%.

The payoff is the changed key. Keyspaces (Vitess's name for a logical group of data that shards together) now scale roughly by users, teams, and channels. Message data is sharded by channel id rather than workspace, so the largest customer's load spreads across the fleet instead of piling onto one host.

THE CHANGED KEY

The post's own farewell is 'say goodbye to only sharding by team, and to team hot-spots.' The proof arrived unscheduled. In March 2020, COVID moved the world's work into Slack and query rates rose 50% in one week, absorbed by splitting the busiest keyspace horizontally with Vitess's splitting workflows. By the post's own accounting, the old architecture would have been unable to scale at all for the largest customers, leading to downtime. The same foundation then carried features the migration never anticipated: Slack Connect's cross-team data, and international data residency running Vitess clusters in six regions.