Before the Wraparound: Sharding Postgres at Notion

Notion's data model revolves around the block — every wiki page, project tracker, and Pokédex is trees of blocks, each a row in one table — and by mid-2020 billions of them had accumulated in a Postgres monolith that had served dutifully through five years and four orders of magnitude of growth. The strain showed the way soft limits always do: engineers on-call woke to database CPU spikes, and simple catalog-only migrations became unsafe and uncertain.

The inflection point was specific: the Postgres VACUUM process began to stall consistently, preventing the database from reclaiming disk space from dead tuples. Disk can be bought; the real threat behind a stalling vacuum was transaction-ID wraparound — the safety mechanism in which Postgres stops processing writes entirely to avoid clobbering existing data. For a product whose every keystroke is a write, wraparound was existential. The team's own aside names the deeper truth: query performance and upkeep processes often degrade well before a table reaches its maximum hardware-bound size, so vertical scaling — 'playing Cookie Clicker with the RDS Resize Instance button' — was not a viable long-term strategy even with the budget for it.

The timing carried its own tension. The aughts' blog canon warned loudly against premature sharding — maintenance burden, application-code constraints, architectural path dependence, and (the post's footnote adds) constraining the product model before the business has defined it. Notion internalized those warnings and waited — and the waiting had a price that shaped the entire migration: by the time sharding became unavoidable, the monolith was too strained to carry extra load, which ruled out convenient tools and forced frugality onto every step that followed.

Notion chose application-level sharding — their own partitioning scheme with queries routed from application logic — over packaged solutions like Citus for Postgres or Vitess for MySQL. The packages appeal in their simplicity and cross-shard tooling, but their clustering logic is opaque, and Notion wanted control over the distribution of its data. (DynamoDB was considered and deemed too risky; bare-metal NVMe Postgres was rejected for the maintenance cost of backups and replication.)

Three design decisions define the scheme. First, what to shard: every table transitively reachable from block via foreign keys, sharded together — because if a block lived on one host and its comments on another, transactionality guarantees that stop at a single host could leave a deleted block with orphaned comment updates. Related data that must commit together stays together. Second, the partition key: workspace ID. Notion is a team product; every block belongs to exactly one workspace, and users query within one workspace at a time, so partitioning the workspace-UUID space into uniform buckets avoids most cross-shard joins. Third, capacity: at least 60K total IOPS of demand, self-imposed bounds of 500 GB per table and 10 TB per physical database, and a bill that scales linearly. The result: 480 logical shards — each a Postgres schema holding one copy of every sharded table — spread 15 per database across 32 physical databases. The number 480 was chosen for its factors: divisible by 2, 3, 4, 5, 6, 8, and every useful count up to 240, it lets the fleet grow 32 → 40 → 48 hosts in increments while keeping shards uniform. A power of two like 512 would force doubling the fleet at every step. Pick values with a lot of factors. They also chose separate schema-qualified tables over Postgres native partitioning, keeping a single source of truth for routing — workspace ID resolves directly to database and schema in application code, with no second routing layer inside Postgres.

The migration followed a four-phase framework: double-write, backfill, verify, switch over. For double-writing, three options were weighed — writing directly to both databases (too flaky for a critical-path store: either write failing breeds inconsistency), Postgres logical replication (the tidy choice — which couldn't keep up with the block table's write volume during its initial snapshot step), and an audit log with a catch-up script, which won: every write to a migrating table is journaled, and a catch-up process replays the journal onto the shards, making modifications as needed — including backfilling the workspace-ID partition key on the fly, because the strained monolith couldn't afford a column backfill. A reverse audit log was built and tested too, ready to replay shard-side writes back onto the monolith if the switchover had to be undone; it was never needed, and it was never optional. The backfill of historical data ran on a 96-CPU m5.24xlarge for roughly three days, comparing record versions before writing so newer updates were never clobbered — run catch-up and backfill in any order and the shards converge on the monolith.

Verification got the discipline the migration's stakes demanded. A script compared sampled UUID ranges between monolith and shards, since a full scan was prohibitive. Dark reads — a flag fetching from both old and new databases, comparing, discarding the sharded copy, and logging discrepancies — bought confidence at the price of API latency. And the post's quietest, sharpest rule: the migration and verification logic were implemented by different people, because one person makes the same mistake twice and calls it a passing check. The switchover itself was five minutes of scheduled downtime, gated on the catch-up script draining the double-write backlog — and users noticed the speed improvement unprompted.