Every Ceiling at Once: How Canva Left MySQL for DynamoDB

Canva's media service stores the identity, ownership, status, and content metadata for every piece of media. It is read-heavy, with most reads for recently created media. For years it was a thin layer over MySQL on Amazon's managed database service (RDS), scaled up, with slightly-stale replica reads. Then its largest tables hit every ceiling at once: schema changes that took six weeks, replication and storage size limits, restart downtime, and a 2TB cap on table files. With the number of media nearing a billion in mid-2017 and doubling, Canva bought time with a few temporary fixes (JSON metadata, denormalizing, ID-range sharding) while migrating live to DynamoDB. For each change, a tiny 'this media changed' message let a worker copy that media's latest version from MySQL into DynamoDB. Hot data moved first, results were compared against MySQL in production, and the cutover ran from a rehearsed checklist with zero downtime, and it costs less to run than the RDS it replaced. Today: 25+ billion media, 50 million more daily.

Interactive

Grow the media count and watch the ceilings light up one after another (schema-change weeks, replication caps, storage limits) then apply the temporary fixes that buy time without removing the walls. Run the live migration: hot data first, a list-by-user query that must wait for 100%, a dual-read comparison that catches the bug early, and a rehearsed, flag-guarded cutover that lands in silence.

Open the visualization ↓

Problem

The media service's shape explains its scale. For each media it stores the ID, the owning user, library membership, external source information, status, a large set of content metadata (title, artist, keywords, color), and references to the underlying files. Reads dominate writes, media are rarely changed after creation, and most reads are for recently created media, with the stock library the exception. Like most of Canva's services, it began as a thin layer over MySQL on RDS, scaled first with bigger instances, then with slightly-stale reads from MySQL replicas.

The cracks arrived through the schema. Changes on the largest media tables began taking days, and MySQL's built-in way of altering tables (online DDL) slowed performance so badly it couldn't run under user traffic at all. GitHub's gh-ost, an open-source migration tool, restored safe online schema changes. Then the walls arrived in formation:

  • MySQL 5.6's replication had a hard rate limit, capping how fast writes could reach the read replicas.
  • Even with gh-ost, schema migrations eventually stretched to six weeks, blocking feature releases.
  • The RDS storage volume was nearing its 16TB limit, and each size increase measurably raised I/O latency, which led to slow user requests.
  • Serving production traffic needed a hot cache in memory, so restarts and upgrades meant accepting downtime.
  • Because the RDS instances were built from snapshots using the older ext3 disk format, MySQL table files were hard-capped at 2TB.
NOT ONE WALL, MANY AT ONCE
Walls lighting at once as media grows: some are MySQL's own limits, the rest the rented host's.
As media grew, Canva's largest tables hit several hard limits at once. Some were MySQL's own (six-week schema changes, a replication cap); the rest belonged to the rented host: the storage volume's cap, restart downtime, and a disk-format table-file limit.

By mid-2017 the number of Canva media approached one billion and was still growing exponentially. The team was clear about its constraint: it strongly preferred incremental steps that kept everything scaling, rather than betting the company on one unproven technology. That meant MySQL had to keep growing while its replacement was investigated, prototyped, and proven against the real workload.

6 weeks
per schema migration on the largest media tables even with gh-ost, blocking feature releases
2TB
hard cap on MySQL table files, inherited from ext3 snapshot provenance on RDS

Solution

Runway came first, through a series of temporary fixes that each bought more time on MySQL:

  • The most-often-changed part of the schema, the content metadata, moved into a single JSON column the service managed itself, so changing it no longer needed a slow schema change.
  • Tables were denormalized (related data folded together) to cut lock contention and joins, foreign-key checks were dropped, and repeated values like S3 bucket names were shortened.
  • At the very end, a deliberately simple scheme split media by ID range across servers, dodging the 2TB file limit and the replication cap. It was tuned for the common case, looking up media by ID when loading a design, and paid for it with slow scatter-gather (asking every server) on everything else, like listing a user's media.

In parallel, the team investigated and prototyped long-term options. With a short runway, a preference for managed services, earlier experience running simpler DynamoDB workloads, and a working prototype, DynamoDB became the tentative target. It still had to be proven against the real workload, with a migration that would touch no user and switch over with zero downtime.

The replication design is the post's quiet masterpiece. Instead of writing every change to both MySQL and DynamoDB at once, replaying an ordered log, or using AWS's migration service, Canva put tiny messages on a queue (SQS). Each recorded only that a given media was created, updated, or read, never the change itself. A worker takes a message, reads that media's current state from the MySQL primary, and writes it to DynamoDB if needed. Because the state is always re-read fresh from the source of truth, the messages can be reordered, retried, paused, or slowed with no effect on correctness, and nobody has to write a fragile log parser.

THE QUIET MASTERPIECE
A content-free message makes a worker re-read the state from MySQL and write to DynamoDB. Reorder and retry are free.
Each queue message says only that one media changed, never what changed. A worker re-reads that media's state from MySQL and writes it to DynamoDB. Because the truth is always fetched fresh, messages can be reordered, retried, or paused safely.

Two queues set the priority: creates and updates on a fast queue, reads on a slow one, with workers emptying the fast queue first, so the data that mattered most stayed the most up to date. To copy the older media, a background scan went through them newest-first (matching how they are read), adding each to the slow queue only when that queue was nearly empty, so it never overloaded anything. This took load off MySQL as soon as the busiest media had moved.

MOVE THE HOT DATA FIRST
A high-priority queue (creates/updates) drains before a low-priority one (reads); a newest-first scan feeds it, so hot data moves first.
Live creates and updates take a high-priority queue, live reads a low one; workers drain the high queue first. A scan walks older media newest-first, feeding the low queue only when it is nearly empty, so the busiest move first.

Proof ran in production. A dual-read comparison served every read from MySQL while checking it against the DynamoDB version, until the replication bugs it surfaced were fixed. Then slightly-stale single-media reads moved to DynamoDB, with a MySQL fallback for the few not yet copied. Some queries don't ask for a media by ID, like 'all media owned by a user'. Those couldn't be answered from DynamoDB until every media had been copied, so they stayed on MySQL until the scan finished, then switched the same careful way.

The write cutover was the riskiest step, and it was wrapped accordingly. New code used all-or-nothing writes (either the whole change is saved or none of it is) and conditional writes to keep the same guarantees as before. Around it went a full test matrix on both the old and new versions, local and end-to-end runs, and a checklist tied to a flag that could send reads back to MySQL within seconds, rehearsed through development and staging. The production cutover was seamless: no downtime, no errors, and median and p95 (the slow-request measure) latency both improved.

THE WHOLE MIGRATION, IN ORDER
Replicate hot data, dual-read compare, move reads to DynamoDB (MySQL fallback), then switch writes; a flag reverts to MySQL.
Copy data live, hot media first. Compare both databases to catch bugs, then move reads to DynamoDB (MySQL as fallback). Only then switch writes, behind a rehearsed checklist. A flag reverts to MySQL in seconds, so every step is safe.

The lessons are printed as commands: be lazy (know your access patterns, migrate the busy data first), do it live, and test in production. Five years on, monthly active users have more than tripled, DynamoDB has autoscaled through all of it at lower cost than the RDS it replaced, and the service holds more than 25 billion media with 50 million arriving daily.

25B+
media stored today with 50M uploaded daily, on DynamoDB that autoscaled through a tripling of monthly active users at lower cost than the RDS it replaced

Tradeoffs

  • Managed convenience carries managed ceilings. Several of the walls weren't MySQL's; they belonged to the rented host: the 16TB storage-volume cap, the 2TB table-file limit from the old disk format, upgrades that cost downtime because the in-memory cache couldn't start cold. Renting the database's host means the host's limits quietly become your schema's limits, and they arrive with less warning than slow queries do.
  • The temporary fixes were real engineering, but everyone knew they had an expiry date. Putting the content metadata in a JSON column let the service change that data on its own, without a slow schema change. Denormalizing and dropping foreign keys traded relational tidiness for less lock contention, and ID-range sharding served the common by-ID lookup while making everything else slow. Each fix bought time by shaping the database more tightly around the few queries it actually ran. That was an early glimpse, still in relational form, of the NoSQL design the service was moving toward.
  • Those tiny 'X changed' messages from the migration, which behindscale calls content-free change events, trade extra reads for the freedom to reorder. Because a message carries only an identity, not the data, a worker has to read the truth from MySQL for every message, which is extra load on the very system being rescued. In exchange, reordering, retrying, pausing, and slowing down are all safe, and nobody has to write a fragile log parser. Putting writes on the fast queue and reads on the slow one is the team choosing to let reads go a little stale, since that is where staleness costs the least.
  • Migrating by access pattern means features come back one query shape at a time. Copying the hot data first took load off MySQL soonest and made recently created media usable soonest, while any query that doesn't ask by ID, like listing a user's media, had to wait until everything had been copied. A '90% replicated' number hides this: a new datastore doesn't light up all at once, it lights up query by query.
  • The confidence at cutover was built, not wished for. The old guarantees were kept with all-or-nothing writes (the whole change saves or none of it does) and conditional writes. Both versions ran the full test suite, the checklist was rehearsed in lower environments, and a flag could roll back in seconds. The result, zero downtime and no errors with latency actually improving, only looks like luck if you skip the paragraph listing all that preparation.
  • Leaving relational came with a bill, and the team lists it plainly. Changing the data's shape or backfilling now means writing and carefully testing special migration code. Ad-hoc SQL queries are gone; that need is now met by streaming changes into a separate data warehouse. And the multi-column indexes DynamoDB needs still have to be built by hand. The post is candid at the end: facing the same problem today, the team would seriously consider a hosted NewSQL database like Spanner or CockroachDB. This answer was right for 2017's limits, not right forever.

Patterns in this article

  • Content-Free Change Events

    This is the migration's quiet masterpiece: queue messages that record only that a media was created, updated, or read, never the content, with workers re-reading the current state from the MySQL primary and writing DynamoDB. Because the state always comes fresh from the source of truth, reordering, retries, pauses, and throttling are all safe, and the ordered-log alternative (with its hand-written log parser) never has to exist. The post is explicit about the options it rejected: writing to both stores, replaying an ordered log, and AWS's migration service.

  • Hot-Data-First Migration

    The post's first lesson, 'be lazy' (know your access patterns and migrate the busy data first), made mechanical. Recently created, updated, and read media replicated ahead of the archive, with a newest-first scan feeding the backlog under backpressure, so load came off the strained MySQL cluster as early as the hot set allowed. The honest cost is that capability arrives per access pattern: queries without an ID had to wait for the scan to finish.

  • Universal Staged Rollout

    The migration end to end: a dual-read comparison in production until replication bugs were fixed, slightly-stale reads with a fallback for stragglers, and test matrices on both implementations for the write cutover. A rehearsed run book and a flag that priced rollback at seconds carried it through development and staging before production. It is another migration in this collection where nothing cut over on faith, the whole risk retired in stages before the irreversible step.

Also solving this

Other systems in behindscale's Single-table scaling ceiling class: