How Discord Indexes Trillions of Messages
Discord redesigned its message search infrastructure to handle trillions of messages, replacing a 2017 architecture that had served well for eight years but had reached its limits. The redesign preserves the original application-layer-sharding decision while replacing every other major component: Redis queues become PubSub for guaranteed delivery, two large Elasticsearch clusters become forty smaller ones grouped into logical cells, single-cluster bulk indexing becomes destination-batched routing, and outlier guilds approaching Lucene's 2-billion-document limit get dedicated cells with multi-shard indices.
Kill the same node in 2017 and in 2025 and compare the blast radius — then trace a message through both eras to find the one routing decision that never changed.
Problem
Discord's original 2017 search architecture (documented in https://discord.com/blog/how-discord-indexes-billions-of-messages) was built around a single architectural insight: route messages in application code to a pool of smaller Elasticsearch clusters, rather than letting Elasticsearch shard internally. This worked extremely well. The system grew from launch to roughly 26 billion documents across two Elasticsearch clusters with 14 nodes total, with median latencies that held steady as the library grew.
By 2024, the same architecture had accumulated five specific failure modes that none of its original design decisions could have anticipated. None of these were design errors — they were the boundaries of choices that were correct at the original scale.
The indexing queue dropped messages under sustained pressure. When Elasticsearch nodes failed, the queue would back up; once the Redis the pipeline depended on hit CPU saturation, messages began silently dropping. The original design treated Redis as a lightweight buffer, which it is — but Discord's growth had turned the indexing queue into a workload Redis was no longer the right tool for.
Bulk indexing was fault-intolerant in a way that scaled poorly with cluster size. The 2017 design batched messages from the queue and sent them as bulk operations to Elasticsearch, with each batch potentially fanning out to many nodes (one bulk operation of 50 messages might touch 50 different nodes in the cluster). Elasticsearch's bulk-operation semantics consider the entire operation failed if any single message fails. With a 100-node cluster, a single failed node meant approximately 40% of bulk operations would fail — turning single-node failures into widespread indexing degradation.
Large clusters had cluster-state coordination overhead that grew faster than capacity. As clusters scaled past 200 nodes, the master node's coordination work began causing OOM crashes, which then caused indexing failures, growing backlogs, query timeouts, and the kind of cascading failure modes that operationally were difficult to recover from quickly.
There was no good path to software upgrades or rolling restarts. The 200+ node clusters required graceful node-by-node restarts that would have taken prohibitively long. The team was forced to run legacy OS and Elasticsearch versions. The most consequential moment came with the log4shell vulnerability: patching required taking the entire search system offline for a maintenance window, because no rolling-restart strategy was available.
Finally, Lucene's MAX_DOC limit — approximately 2 billion documents per index — was reachable by indices containing the messages of Discord's largest guilds. Once an index hit MAX_DOC, all indexing operations to it failed. The original workaround was to identify spam guilds and delete them. This was acceptable as a short-term tactic but didn't scale: legitimate communities were beginning to accumulate similar message counts.
The redesign needed to address all five problems simultaneously, while preserving what worked about the original design — particularly the application-layer routing decision, which had given Discord exactly the operational control over sharding it needed.
Solution
The 2025 redesign keeps Discord's foundational architectural decision intact: application-layer sharding, with the routing logic owning which Elasticsearch cluster and index receives each message. Every other major component was replaced.
Elasticsearch now runs on Kubernetes via the Elastic Cloud on Kubernetes (ECK) operator. This solves the rolling-restart problem at its root — OS upgrades happen automatically, and the operator exposes ergonomic tooling for safely rolling clusters through configuration changes. The team is no longer locked into legacy versions because they can no longer afford the downtime of upgrading.
The biggest architectural shift is the move from two large clusters to roughly forty smaller ones, grouped into logical 'cells.' Each cell is a set of clusters dedicated to a specific use case. The 'guild-messages' cell holds messages sharded by guild_id (the same sharding key the 2017 architecture used). A new 'user-dm-messages' cell holds DMs sharded by user_id (a different sharding key that enables cross-DM search — a long-requested feature that the original guild_id sharding couldn't support without prohibitive query fanout). A 'BFG' cell holds messages from 'Big Freaking Guilds' that have approached or exceeded Lucene's MAX_DOC threshold.
Within each cluster, the topology is now resilient to zonal failure: three master-eligible nodes (one per zone), at least three ingest nodes (one per zone), and data nodes with shard allocation awareness ensuring primary and replica live in different zones. This is the kind of design that becomes possible only when clusters are small enough that running three master-eligible nodes is affordable per-cluster.
The indexing queue migrated from Redis to Google Cloud PubSub. PubSub provides guaranteed message delivery and tolerates large backlogs without dropping. Elasticsearch failures now cause indexing slowdowns rather than message loss — which is the correct failure mode for a search-update queue. The team mentions adopting PubSub for other Discord use cases after this migration, which is itself meaningful: the choice generalized.
The bulk-indexing fault-tolerance problem was solved by a routing layer between PubSub and Elasticsearch. A Rust service consuming PubSub messages spawns a tokio task per destination (cluster + index), with each task accumulating messages keyed to that destination before issuing bulk operations. The result: each bulk operation now targets a single Elasticsearch node. A single-node failure affects only the bulk operations destined for that node, not the entire indexing fleet. The architectural lesson is general — when bulk operations have fanout costs, batch by destination before issuing them.
For outlier guilds approaching MAX_DOC, the BFG cell takes a different approach than ordinary cells. Most Elasticsearch indices have a single primary shard (Discord's standard, optimal for query performance because all messages for an index live on one node). BFG indices have multiple primary shards, accepting the coordination overhead of multi-shard queries in exchange for being able to scale past the single-shard MAX_DOC limit. The reindexing flow handles the migration gracefully: dual-index new messages to both the old and new index, historically reindex the old data, switch query traffic when the new index is verified, then clean up.
The production results, published in the post: trillions of messages indexed (up from billions), 2x indexing throughput, median query latency from 500ms to under 100ms, p99 latency from 1 second to under 500ms, and forty Elasticsearch clusters running automated upgrades and rolling restarts with no service impact.
Tradeoffs
- Application-layer sharding is preserved through the redesign, which means Discord still owns the operational complexity of routing logic. The cell architecture made this more complex, not less — there are now more destinations to route to, with logic for choosing which cell handles which sharding key. The team accepts this in exchange for the failure-isolation properties they couldn't get from letting Elasticsearch shard internally.
- Running forty smaller clusters instead of two large ones means roughly twenty times more cluster-management surface area. The ECK operator handles much of this, but the team must understand and operate forty cluster topologies, monitor forty masters, debug forty potential failure modes. The benefit is that no single cluster's coordination overhead grows beyond what its master node can handle; the cost is that operational tooling and dashboards must work across many clusters rather than depth-debugging two.
- PubSub provides guaranteed delivery and tolerates large backlogs, but it introduces a cross-cloud dependency on Google Cloud's PubSub service (Discord runs on GCP, so this is bounded — but the search system is now coupled to a managed service with its own pricing model and operational characteristics). The team accepts this because the failure mode of Redis under pressure (silent message drops) was unacceptable for a system that's the source of truth for search indexing.
- The destination-batched routing adds latency to indexing: messages now wait for a batch to fill (or a timeout to fire) before being sent to Elasticsearch. For Discord's use case — where 'searchable within a minute' is acceptable — this is fine. For a system that needed real-time indexing, the batching strategy would be wrong.
- Cross-DM search required indexing each DM message twice (once per recipient's user_id-sharded index), roughly doubling storage cost for DM messages. The team accepts this because the alternative — fanning out search queries across every channel a user has DMed in — was operationally untenable. Indexing twice at write time is much cheaper than fanning out at query time for read-heavy workloads.
- The BFG architecture introduces a parallel index lifecycle that the rest of the system doesn't need to know about. The dual-indexing-then-cutover process during BFG reindexing means there's a temporary period where the same messages exist in two places, with the routing layer responsible for ensuring consistency. The team accepts this complexity for the small number of guilds that need it; most guilds remain on the simpler single-shard model.
Patterns in this article
- Application-Layer Sharding
Discord's foundational decision in 2017, preserved through the 2025 redesign. The routing logic lives in application code, not in Elasticsearch's internal sharding. This gives Discord control over which clusters and indices receive each message — a control that proved essential when the time came to evolve the architecture. The pattern's value is most visible in retrospect: the redesign was possible *because* sharding was already an application-level concern that the team could refactor without depending on Elasticsearch's internal coordination.
- Cell Architecture
The 2025 redesign's grouping of smaller Elasticsearch clusters into logical cells, each dedicated to a specific use case (guild messages, user DMs, Big Freaking Guilds). The architectural unit shifts from 'cluster' to 'cell of clusters,' which provides failure isolation at a more useful granularity and enables per-use-case optimization. Discord's cell architecture is one of the cleanest published examples — the cells are visible in the architecture, the use cases are distinct, and the BFG cell is a textbook 'one cell per outlier class' implementation.
- Queue with Guaranteed Delivery
The Redis-to-PubSub migration. Discord's original Redis indexing queue dropped messages under sustained pressure — a 'buffer' rather than a 'queue.' PubSub provides guaranteed delivery and tolerates large backlogs, which means downstream failures now manifest as slowdowns rather than data loss. The pattern's principle: if your queue's failure mode is data loss, you don't have a queue, you have a buffer. Real queues persist.
- Batched Routing by Destination
The Rust+tokio routing layer that consumes PubSub messages, groups them by destination (cluster + index), and issues each bulk operation to a single node. This replaced the previous bulk-indexing model where one batch could fan out across many nodes (and a single failed node would fail an outsized fraction of operations). The general pattern is preserving the throughput benefit of bulk operations while keeping each bulk operation's fault domain narrow.
- Fault Isolation
Cell architecture is fault isolation at the cluster-group level. A failure in the BFG cell doesn't affect the user-dm-messages cell. A bad deploy of routing logic for guild messages doesn't impact DM search. The 2025 redesign sets up a hierarchy of isolation boundaries (cluster, cell, use case) that the original two-cluster architecture couldn't express. This is the same pattern Airbnb applies in their monitoring infrastructure, now applied at a different layer.