Before the Wraparound: Sharding Postgres at Notion
In Notion, every piece of content (a paragraph, heading, or image) is a 'block', and all of them live as rows in one giant Postgres table. That block table carried five years and four orders of magnitude of growth before its volume began defeating the single database beneath it: VACUUM (Postgres's housekeeping that reclaims space from dead rows) stalled, and behind it waited transaction-ID wraparound, a safety mechanism that halts all writes and would be fatal for a write-heavy product. The answer was application-level sharding, chosen over off-the-shelf tools like Citus and Vitess so Notion could control its data placement: 480 logical shards (picked for its factors) across 32 physical databases, partitioned by workspace ID, with related tables kept together so transactions never cross hosts. The migration ran four phases: double-write via an audit log, a three-day backfill on 96 CPUs, sampled verification and dark reads written by different people than the migration, and a five-minute switchover the team says could have been zero.
Migrate the monolith before the wraparound: pick your double-write strategy, backfill, verify — and see what your catch-up lag costs at the switchover. Then try scaling 512 shards.
Problem
Notion's data model revolves around the block, its atomic unit of content: every paragraph, heading, image, and to-do a user creates is a block, so every wiki page, project tracker, and Pokedex is a tree of blocks, and each block is a row in a single Postgres table. By mid-2020 billions of them had accumulated in a Postgres monolith that had served through five years and four orders of magnitude of growth. The strain showed the way soft limits always do: on-call engineers woke to database CPU spikes, and simple schema changes became unsafe and uncertain.
The inflection point was specific: the Postgres VACUUM process began to stall consistently. VACUUM is the housekeeping that reclaims space from dead rows (old row versions no longer visible to any query), and when it stalls that space is never reclaimed. Disk can be bought; the real threat behind a stalling vacuum was transaction-ID wraparound, the safety mechanism in which Postgres stops processing writes entirely to avoid overwriting existing data. For a write-heavy product like Notion, where editing constantly writes to the database, wraparound was existential. The team's own aside names the deeper truth: query performance and upkeep processes often degrade well before a table reaches its maximum hardware-bound size, so scaling vertically (just renting a bigger and bigger single instance, which the post memorably likens to clicking a video-game upgrade button over and over) was not a viable long-term strategy even with the budget for it.
The timing carried its own tension. The blog canon of the 2000s warned loudly against sharding too early: maintenance burden, constraints imposed on application code, architectural path dependence, and (the post's footnote adds) locking in the product model before the business has defined it. Notion took those warnings to heart and waited, and the waiting had a price that shaped the entire migration: by the time sharding became unavoidable, the monolith was too strained to carry any extra load, which ruled out the convenient tools and forced frugality onto every step that followed.
Solution
Notion chose application-level sharding (their own partitioning scheme, with queries routed from application code) over off-the-shelf tools like Citus for Postgres or Vitess for MySQL, which automatically spread a single database across many machines and hide the mechanics from you. The packages appeal in their simplicity and cross-shard tooling, but their clustering logic is opaque, and Notion wanted control over how its data was distributed. (DynamoDB was considered and judged too risky; bare-metal NVMe Postgres was rejected for the maintenance cost of backups and replication.)
Three design decisions define the scheme. First, what to shard: every table reachable from block through foreign keys, sharded together, because if a block lived on one host and its comments on another, the guarantee that a set of changes all commit together (which holds only within a single host) could leave a deleted block with orphaned comment updates. Related data that must commit together stays together. Second, the partition key (the value used to decide which shard a row goes to): workspace ID. A workspace is a single team's whole Notion account; Notion is a team product, every block belongs to exactly one workspace, and users query within one workspace at a time, so splitting the workspace-ID space into uniform buckets avoids most queries that would otherwise span shards. Third, capacity: at least 60K total IOPS (disk operations per second) of demand, self-imposed limits of 500 GB per table and 10 TB per physical database, and a bill that scales linearly. The result: 480 logical shards, each a Postgres schema holding one copy of every sharded table, spread 15 per database across 32 physical databases. The number 480 was chosen for its factors: divisible by 2, 3, 4, 5, 6, 8, and every useful count up to 240, it lets the fleet grow 32 to 40 to 48 hosts in steps while keeping the shards evenly spread. A power of two like 512 would force doubling the fleet at every step, so they picked the value with many factors. They also chose separate schema-qualified tables over Postgres's built-in partitioning, keeping one source of truth for routing: a workspace ID resolves directly to a database and schema in application code, with no second routing layer inside Postgres.
The migration followed a four-phase framework: double-write, backfill, verify, switch over. For double-writing, three options were weighed. Writing directly to both databases was too flaky for a critical store (either write failing breeds inconsistency). Postgres logical replication (its built-in feature for streaming changes from one database to another) was the tidy choice, but it couldn't keep up with the block table's write volume during its initial snapshot step. So the winner was an audit log with a catch-up script: every write to a migrating table is journaled, and a catch-up process replays the journal onto the shards, making changes as needed, including filling in the workspace-ID partition key on the fly, because the strained monolith couldn't afford to backfill that column directly. A reverse audit log was built and tested too, ready to replay shard-side writes back onto the monolith if the switchover had to be undone; it was never needed, and it was never optional. The backfill of historical data ran on a 96-CPU m5.24xlarge for roughly three days, comparing record versions before writing so newer updates were never overwritten; run catch-up and backfill in any order and the shards still converge on the monolith.
Verification got the discipline the stakes demanded. A script compared sampled ranges of the UUID space between monolith and shards, since a full scan was too expensive. Dark reads (a flag that fetches from both old and new databases, compares them, discards the sharded copy, and logs any mismatch) bought confidence at the cost of some API latency. And the post's quietest, sharpest rule: the migration and the verification were written by different people, because one person tends to make the same mistake twice and then call it a passing check. The switchover itself was five minutes of scheduled downtime, gated on the catch-up script draining the double-write backlog, and users noticed the speed improvement unprompted.
Tradeoffs
- Waiting until sharding was unavoidable meant migrating from a position of weakness, and the post's first hindsight lesson is simply 'shard earlier.' The strained monolith couldn't absorb the load of logical replication's snapshot or a partition-key column backfill, which forced the custom audit-log machinery and the on-the-fly key backfilling. The essays warning against sharding too early are right about the costs of moving too soon, but they say nothing about the costs of moving too late: the longer you wait, the fewer migration tools you can still afford to use.
- Application-level sharding buys control and pays for it in ownership, forever. Rejecting Citus and Vitess for their opaque clustering logic means Notion now owns routing, rebalancing, cross-shard tooling, and every future re-shard in its own application code, exactly the added application-code constraints the old warnings named, accepted here with eyes open. The 480-factors choice is this tradeoff done well: because Notion owned the scheme, it could build the future flexibility in from the start, and picking a number with many factors is what let the fleet grow in small even steps instead of by doubling.
- Partitioning by workspace ID lines the shards up with the product's natural transaction boundaries, but it also inherits how unevenly the product's load is spread. Keeping everything transitively related to block together preserves single-host transactions and kills most cross-shard queries, but the post itself notes that one large enterprise customer can generate more load than many personal workspaces combined; uniform buckets of workspace IDs are uniform in count, not in weight. The scheme's honest bet is that this hot-workspace skew stays manageable. And the footnoted danger of sharding too early cuts here too: shard by the wrong entity, and a later product pivot turns the partition key into architectural debt.
- The audit log was the right tool and the expensive one. Logical replication is built in, tested by the whole world, and needs no custom code; the audit log is custom-built code sitting on the critical write path, plus a catch-up script the whole migration's correctness leans on, plus a reverse log for the road back. Notion paid the build cost because their situation (the write volume, the missing partition key) ruled out the standard tool. Which tool you can use is set by how strained the system already is, not by which one is best in theory.
- Verification that can actually catch a migration's errors has to be independent of the migration itself: sampled checks, dark reads, and separate authors. Sampling trades certainty for something you can actually run; dark reads trade latency for confidence; separate implementers trade extra staffing for the independence that makes their mistakes uncorrelated. Each is a deliberate weakening of proof in exchange for proof you can afford, and the two-author rule is the cheapest strong guarantee in the whole post: what lets a bad migration pass its own tests is not missing checks but the same blind spot showing up in both the migration and the check.
- Five minutes of downtime turned out to be a choice, visible as one only in hindsight. The switchover was gated on the catch-up script draining, and the team's own accounting says one more week spent optimizing it to a sub-30-second catch-up might have allowed a hot swap at the load balancer with no downtime at all. Against months of urgent work, a week to turn five minutes into zero looks obviously worth it, and that is the lesson: the length of a downtime window is something you can shorten by spending engineering effort on it, not a fixed cost you simply accept, and the moment to work out whether it's worth shortening is before you schedule the maintenance, not after.
Patterns in this article
- Application-Layer Sharding
Notion names the approach outright: build your own partitioning scheme and route queries from application code, chosen over packaged tools like Citus and Vitess because their clustering logic is opaque and Notion wanted control over how its data was distributed. Three companies in the library have now made this same call over different starting points: Discord moving off Elasticsearch, Figma adding a query-routing service in front of Postgres, and Notion routing directly in application code. The shared reason is the same each time: owning the routing is worth the ownership cost when the packaged option hides how your data is placed.
- Shard-Key Colocation
Both halves of the pattern sit in one post. Shard every table reachable from block so that rows which must change together live on the same host (a single database is as far as a commit-together guarantee reaches), and partition all of it by workspace ID so that a given user's queries land on one shard. The example the post gives is the failure case stated plainly: a block deleted on one host while its comments, stranded on another, never get the update.
Also solving this
Other systems in behindscale's Single-table scaling ceiling class: