The Largest Shard Money Could Buy: Slack's Road to Vitess

The original architecture is laid out with unusual completeness. Three MySQL cluster families: shards, holding virtually all customer data — messages, channels, DMs — partitioned by workspace id, thousands of workspaces per shard, all of a workspace's data on one shard so the application connects to exactly one database; a metadata cluster mapping workspace id to shard id, consulted first on every access; and a kitchen-sink cluster for data tied to no workspace, like the app directory. Every shard was at least two MySQL instances in different data centers, active-active with asynchronous replication — both sides taking reads and writes, a hash choosing the preferred side, the other side absorbing retries on failure. The webapp monolith owned all of it: metadata lookup, connection routing, sharding logic.

The post is fair to this design before it buries it: high availability from the dual-write pair, high product velocity from the everything-in-one-place model, minutes-to-diagnosis debuggability, and easy growth — more teams, more shards. Then the bolded question: what if a single team and all of their Slack data doesn't fit our largest shard?

By fall 2016 — hundreds of thousands of MySQL QPS, thousands of sharded hosts — the disadvantages had names. Scale limits: the largest customers' designated shards reached the largest available hardware, regularly hitting what a single host could sustain. One data model: Enterprise Grid and what became Slack Connect challenged the paradigm that a team's data lives on one shard, adding complexity and, in cases, a performance penalty. Hot spots beside waste: unable to spread a large customer's load, Slack ran a few burning shards and a massively underutilized long tail, overprovisioned because splitting shards and moving teams was hard and usage unpredictable. Availability coupling: every core feature needed the team's shard up, so a shard outage was a total Slack outage for every customer on it. And operations: the active-active topology was nonstandard MySQL, demanding significant internal tooling, and — with no replicas in the topology and the app routing directly to hosts — replica reads couldn't be used safely at all.

The fork in the road was evolve-in-place versus replace. The in-place option — teaching the webapp's existing routing layer new keys like channel id — was prototyped seriously and rejected on the evidence: the application logic and the storage layout were deeply coupled (fetching a channel's message count assumed which team's shard held it; multi-workspace organizations were handled by explicitly checking multiple shards), untangling would be slow, and the approach fixed neither operations nor replica usage — the post's long-term verdict being that a surprisingly hot single-team shard would still not be straightforward to scale. The constraint that shaped everything else: stay on MySQL. Thousands of distinct queries, some using MySQL-specific constructs, plus years of operational practice — deployment, durability, backups, warehouse ETL, compliance — ruled out NoSQL (DynamoDB, Cassandra) and NewSQL (Spanner, CockroachDB) as far more disruptive changes for a deliberately conservative infrastructure culture.

Vitess fit the requirements as built: MySQL remains the actual storage and replication engine; sharding lives in the database layer, flexible and growable without application logic; operability comes built in — automated primary failovers, backups, a lock server tracking topology so the application can be blissfully ignorant of where anything lives; and it's open-source Go that Slack could extend, which they did to the point of becoming one of the project's biggest contributors (topology metadata refactored for isolation regions, MySQL compatibility gaps closed, migration and load-testing tools, Prometheus and Orchestrator and xtrabackup integrations).

The migration was staged with the care of a system that cannot stop running: a first end-to-end production use case deliberately small — RSS feeds into channels — that forced the real work of provisioning, service discovery, backup/restore, topology management, and credentials; then a generic backfill system cloning existing tables under application double-writes, with a parallel double-read diffing system proving the Vitess-backed tables behaved identically to the legacy ones before cutover. Three years later: 99% of all MySQL traffic on Vitess, the rest scheduled within two months, at 2.3 million QPS peak — 2 million reads, 300 thousand writes — with 2ms median and 11ms p99 latency.

The payoff is the changed key. Keyspaces now scale roughly by users, teams, and channels — message data sharded by channel id rather than workspace, so the largest customer's load spreads across the fleet instead of concentrating on their designated host; the post's own farewell is 'say goodbye to only sharding by team, and to team hot-spots.' The validation arrived unscheduled: in March 2020, COVID moved the world's work into Slack and query rates rose 50% in one week — absorbed by horizontally splitting the busiest keyspace with Vitess's splitting workflows, where the old architecture, by the post's own accounting, would have been unable to scale at all for the largest customers, leading to downtime. The same substrate then carried features the migration never anticipated: Slack Connect's cross-team data, and international data residency running Vitess clusters in six regions.