Batched Routing by Destination

throughput

Definition

Batched routing by destination is a strategy for issuing bulk operations against partitioned downstream systems without paying the fanout cost of cross-partition batches. Incoming items arrive in an interleaved stream (orders by region, log lines by service, messages by tenant); a routing layer groups items by their downstream destination (database shard, search cluster, partition, node) and issues each bulk operation against a single destination. The result: each bulk operation has a narrow fault domain, single-destination failures no longer cause widespread batch failures, and the downstream system can apply per-destination optimizations.

The failure mode the pattern addresses is fanout amplification of single-point failures. When a bulk operation contains items destined for many backend nodes — say, 50 messages going to 50 different shards — the operation as a whole is considered failed if any single destination fails. A single failed node causes a disproportionate fraction of bulk operations to fail. With 100 evenly-distributed destinations and 50-item batches, a single-node failure causes roughly 40% of bulk operations to fail (the math is approximately 1 - (99/100)^50). The failure mode scales badly: more destinations make it worse, not better.

The fix is to require each bulk operation to target exactly one destination. The routing layer reads the stream, examines each item's destination key, and dispatches the item to a per-destination collector. Each collector accumulates items until it has enough to issue a useful bulk operation (or until a timeout fires), then sends the batch to that destination only. Single-destination failures now affect only their own batches; multi-destination failures still exist but are bounded by the destinations affected, not by the cross-product of destinations in any batch.

The implementation pattern is straightforward in any language with first-class concurrency primitives: a dispatcher reads the stream, a per-key task pool collects items into batches, and timeouts force batches to flush even when not full. Discord's 2025 implementation uses Rust tokio tasks with one task per cluster+index destination. The same shape appears in many systems: Kafka producers batching by partition, database connection pooling with per-shard batches, log aggregators batching by destination index.

When it applies

  • Indexing pipelines feeding sharded or partitioned search systems where bulk operations are the throughput-optimal write path but cross-shard batches amplify failure rates
  • Database write pipelines targeting horizontally-partitioned data stores (sharded Postgres, Cassandra by partition key, DynamoDB by partition) where cross-partition writes have higher overhead than single-partition writes
  • Log and metric aggregation systems shipping data to multiple downstream destinations where keeping batches destination-specific improves both throughput and isolation
  • Any bulk-operation API where the operation's failure semantics are 'all-or-nothing per call' and the cost of a partial-batch failure is high (re-enqueueing, retrying, or surfacing errors to users)
  • Pipelines transitioning from naive batching to fault-tolerant batching as scale grows past the point where cross-destination failure amplification becomes operationally visible

Tradeoffs

  • Batched routing adds end-to-end latency. Items wait in their per-destination collector until the batch fills or the timeout fires. For workloads that need 'searchable within seconds,' this is fine; for workloads where every millisecond of indexing latency matters, the latency added by batching may be the wrong tradeoff.
  • Per-destination state accumulates in the routing layer. With N destinations, the routing layer holds N collectors, each with their own buffer of pending items. Memory cost is real, especially when destinations are numerous and items are large. Discord's 2025 architecture has thousands of destinations (clusters times indices), each potentially holding a batch in flight.
  • Destinations with low traffic spend most of their time waiting for timeouts. A destination that receives 1 item per second with a batch size of 100 and a timeout of 1 second sends 1-item batches most of the time, which defeats the throughput benefit of batching. The pattern works best when traffic distribution is reasonably balanced across destinations or when timeouts are tuned per-destination.
  • The routing layer becomes a critical, stateful, in-line component. If the router dies, in-flight items in collectors are lost (unless the collectors themselves are persistent). Production implementations usually pair this with a durable upstream queue so that router failures can be recovered by replay rather than by data loss.
  • Adding new destinations is a dynamic operation. The router must handle destination discovery, collector creation, and destination retirement gracefully. Static-destination implementations are simpler; dynamic ones are more flexible but require more careful design.

Seen in

  • Discord EngineeringApr 24, 2025

    How Discord Indexes Trillions of Messages

    The Rust+tokio routing layer that consumes PubSub messages, groups them by destination (cluster + index), and issues each bulk operation to a single node. This replaced the previous bulk-indexing model where one batch could fan out across many nodes (and a single failed node would fail an outsized fraction of operations). The general pattern is preserving the throughput benefit of bulk operations while keeping each bulk operation's fault domain narrow.