How Discord Indexes Trillions of Messages
Discord redesigned its message search to handle trillions of messages, replacing a 2017 system that had served well for eight years but had reached its limits. The redesign keeps the one decision that aged well, routing each message to its shard in application code, while replacing everything that routing hands the message to next. The Redis queue becomes Google Cloud PubSub, which never drops messages under load. The two large Elasticsearch clusters become about forty smaller ones grouped into logical 'cells'. Adding messages to the search index in bulk (many at once) learns to group each batch by where it is going. And the few guilds (Discord servers) nearing two billion documents, the hard limit of Lucene (the search engine underneath Elasticsearch), get their own cells with indices split across several shards.
Kill the same node in 2017 and in 2025 and compare the blast radius, then trace a message through both eras to find the one routing decision that never changed.
Problem
Discord's original 2017 search architecture was built around a single idea: route each message in application code to one of a pool of smaller Elasticsearch clusters, rather than letting Elasticsearch split the data up internally. (Discord's 2017 write-up, 'How Discord Indexes Billions of Messages', covers it.) This worked extremely well. The system grew to roughly 26 billion documents across two Elasticsearch clusters, 14 nodes in all, and search stayed fast as the message library grew.
By 2024, that same architecture had run into five specific failure modes that none of its original choices could have anticipated. None of these were mistakes; they were the limits of decisions that had been correct at the original scale. The five:
- The queue of messages waiting to be indexed dropped some of them. When Elasticsearch nodes failed, that queue filled up faster than it could drain, and once the Redis holding it maxed out its CPU, messages were silently lost. Redis was being used as a lightweight holding area, but the queue had grown into a job Redis was no longer the right tool for.
- Messages were indexed in batches, and one bad node failed the whole batch. To index efficiently, workers send 50 messages to Elasticsearch in a single batch, but those 50 can land on 50 different nodes, and Elasticsearch treats the whole batch as failed if even one message fails. On a 100-node cluster, a single dead node failed about 40% of all batches, turning one node's death into a fleet-wide slowdown.
- Coordination overhead grew faster than capacity. Past 200 nodes, the cluster's coordinating 'master' node ran out of memory, which caused indexing failures, growing backlogs, query timeouts, and cascades that were hard to recover from quickly.
- There was no safe way to restart or upgrade. Restarting 200-plus-node clusters node by node would have taken far too long, so the team was stuck on old software. When the log4shell security bug hit, patching meant taking all of search offline for a maintenance window.
- The biggest guilds hit a hard limit. Lucene, the engine under each Elasticsearch index, caps an index at about two billion documents; once a guild's index hit that, all further indexing to it failed. The workaround was to delete spam guilds, but that didn't last, because legitimate communities were reaching the same size.
The redesign had to fix all five at once, while keeping what worked about the original: above all, the application-code routing, which had given Discord exactly the control over sharding it needed.
Solution
The 2025 redesign keeps Discord's foundational decision intact: sharding in application code, where the routing logic decides which Elasticsearch cluster and index each message goes to. Every other major piece was replaced.
Elasticsearch now runs on Kubernetes, managed by the Elastic Cloud on Kubernetes (ECK) operator, a tool that automates cluster operations. This fixes the restart problem at its root: OS upgrades happen automatically, and the operator provides safe, easy tooling for rolling a cluster through configuration changes one node at a time. The team is no longer stuck on old versions to avoid the downtime of upgrading.
The biggest shift is going from two large clusters to roughly forty smaller ones, grouped into logical 'cells'. Each cell is a set of clusters dedicated to one job. The 'guild-messages' cell holds messages split by guild (the same key as 2017). A new 'user-dm-messages' cell holds direct messages split by user, which finally makes 'search all my DMs' possible; splitting by channel before would have meant querying every DM separately. And a 'BFG' cell holds messages from 'Big Freaking Guilds', the ones near or over Lucene's two-billion limit.
Because the clusters are now small, each one can afford a layout that survives losing a whole data-center zone:
- three master-eligible nodes, one per zone, so coordination survives a zone loss;
- at least three ingest nodes, one per zone, to receive and route incoming writes;
- data nodes placed so an index's primary copy and its backup copy live in different zones.
The indexing queue moved from Redis to Google Cloud PubSub. PubSub guarantees delivery and holds large backlogs without dropping anything. So an Elasticsearch failure now just slows indexing down instead of losing messages, which is the right way for a search-update queue to fail. In fact, the team liked PubSub enough to start using it for other jobs across Discord too.
That whole-batch-fails problem was solved by adding a routing layer between PubSub and Elasticsearch. A small Rust service reads from PubSub and keeps a separate lightweight worker for each destination (a cluster-and-index pair), and each worker gathers only the messages headed to its own destination before sending them as one batch. The result: each batch now goes to a single Elasticsearch node. So one node's death affects only the batches headed for that node, not the whole fleet. The general lesson: when sending things in bulk is costly to fan out, group them by destination first.
For the outlier guilds near the two-billion limit, the BFG cell works differently. Most indices use a single primary shard (Discord's default, best for query speed because all of an index's messages sit on one node). BFG indices use several primary shards instead, accepting the extra coordination of querying across shards in exchange for scaling past the one-index limit. A guild is moved onto a bigger index without downtime, in four steps:
- write every new message to both the old and the new index at once;
- copy the old messages across in the background;
- switch search traffic to the new index once it is verified;
- stop writing to the old index and delete it.
The results Discord published: trillions of messages indexed, up from billions, at twice the indexing throughput. Median query latency dropped from 500ms to under 100ms, and the slowest 1% of queries (p99) from 1 second to under 500ms. And forty Elasticsearch clusters now run automated upgrades and restarts with no service impact.
Tradeoffs
- Keeping sharding in application code means Discord still owns the complexity of the routing logic. The cell architecture made that harder, not easier: there are now more destinations to route to, plus logic for choosing which cell handles which key. The team accepts this in exchange for the failure isolation it could never get by letting Elasticsearch shard internally.
- Running forty small clusters instead of two big ones means roughly twenty times more to manage. The ECK operator handles much of it, but the team still has to understand and run forty cluster layouts, watch forty coordinating nodes, and debug forty possible failure modes. The benefit is that no single cluster's coordination load ever grows past what its master node can handle; the cost is that the tooling and dashboards now have to work across many clusters instead of going deep on two.
- PubSub guarantees delivery and holds large backlogs, but it ties search to a managed service (Google Cloud PubSub) with its own pricing and behavior. Because Discord already runs on Google Cloud, this dependency is contained rather than a new outside vendor. The team accepts it because the Redis failure under pressure, silently dropping messages, was unacceptable for the system that is the source of truth for search.
- Batching by destination adds a little delay to indexing: a message now waits for its batch to fill (or a timer to fire) before it is sent to Elasticsearch. For Discord, where 'searchable within a minute' is fine, this is a good trade. For a system that needed real-time indexing, batching this way would be the wrong choice.
- Search-across-DMs required indexing each direct message twice, once in each participant's user index, roughly doubling storage for DMs. The team accepts this because the alternative, fanning a search out across every DM channel a user is in, was operationally hopeless. For read-heavy work, paying to index twice up front is far cheaper than paying to fan out on every query.
- The BFG design adds a second, separate way of managing indexes that the rest of the system can ignore. While a big guild is being moved to a larger index, its new messages are written to both the old and the new index at once. So the same messages briefly live in two places, and the routing layer has to keep them in sync. The team accepts this extra complexity for the small number of guilds that need it; everyone else stays on the simpler single-shard model.
Patterns in this article
- Application-Layer Sharding
Discord's foundational decision in 2017, kept through the 2025 redesign. The routing logic lives in application code, not inside Elasticsearch. This gives Discord control over which clusters and indices each message goes to, control that proved essential when the time came to evolve the architecture. The value is clearest in hindsight: the redesign was possible because sharding was already an application-level concern the team could rework, without waiting on Elasticsearch's internal coordination.
- Cell Architecture
The 2025 redesign groups smaller Elasticsearch clusters into logical cells, each dedicated to one use case (guild messages, user DMs, Big Freaking Guilds). The basic unit shifts from 'cluster' to 'cell of clusters', which isolates failures at a more useful size and lets each cell be tuned for its own job. In Discord's version the cells map directly to distinct use cases, and the BFG cell is a clean example of giving one outlier class its own cell.
- Queue with Guaranteed Delivery
The Redis-to-PubSub migration. Discord's original Redis indexing queue dropped messages under sustained pressure: a 'buffer', not a real 'queue'. PubSub guarantees delivery and holds large backlogs, so downstream failures now show up as slowdowns rather than lost data. The principle: if your queue's failure mode is losing data, you don't have a queue, you have a buffer. Real queues persist.
- Batched Routing by Destination
The Rust routing layer reads PubSub messages, groups them by destination (a cluster-and-index pair), and sends each batch to a single node. This replaced the old model, where one batch could fan out across many nodes and a single failed node would sink an outsized share of operations. The general pattern: keep the throughput win of batching while keeping each batch's blast radius small.
- Fault Isolation
Cell architecture is fault isolation at the level of a group of clusters. A failure in the BFG cell doesn't touch the user-dm-messages cell; a bad deploy of guild-message routing doesn't break DM search. The redesign sets up a ladder of isolation boundaries (cluster, then cell, then use case) that the original two-cluster setup couldn't express. Airbnb applies the same pattern in its monitoring infrastructure, at a different layer.
Also solving this
Other systems in behindscale's Blast radius scales with cluster size class: