Almost Exactly Once: Segment's Billion-Message Dedupe Ledger

The post opens with the pipeline's prime directive — data can be delayed or re-ordered, never dropped — and its standard consequence: at-least-once delivery everywhere, achieved by never considering a message delivered until firmly acknowledged, and retrying otherwise. Segment's internal systems handle the resulting failure modes gracefully with retries, re-delivery, locking, and two-phase commits. The stated exception is the system's front door: clients that send data directly to the public API.

The motivating picture is concrete: booking a hotel from a phone on a bus, the app uploading usage events to Segment's servers, the bus entering a tunnel. Some events have been processed; the response never arrives; the client — correctly — retries and re-sends the same events the server technically already received. Nothing is misbehaving. The client cannot distinguish a lost request from a lost response, and on mobile networks that ambiguity is routine, not exceptional. Segment's server metrics put a number on it: approximately 0.6% of events ingested within a four-week window are duplicates of messages already received. The post is direct about why that matters — for an e-commerce app generating billions in revenue, a 0.6% discrepancy can be the difference between profit and a loss of millions.

And the post is equally direct about the ceiling: achieving anything close to exactly-once delivery requires a bullet-proof design where each failure case is considered as part of the architecture — it cannot be bolted on afterward — and even then it is pretty much impossible to have messages delivered only ever once. The dedupe problem statement is deceptively simple (has_seen(message.id) → discard; else publish and commit, atomically), with a de-duplication window bounding how long identities are remembered. Everything hard lives in two words: performance — deduplicating billions of events at low latency and acceptable cost — and correctness — durably remembering what's been seen across crashes, while never producing duplicates in the output.

Identity comes first: every incoming message is tagged with a unique messageId, generated by the client — a UUIDv4, with the API assigning one only when the client doesn't. The post names the road not taken: no vector clocks, no sequence numbers, because those raise client complexity, and UUIDs let anyone in almost any language send data. The pipeline's exactly-once ambition rests on the humblest possible client obligation.

The architecture is a two-phase Kafka topology. Incoming API calls are split into individual messages and logged to a Kafka input topic for durability and replayability — partitioned by messageId, so the same identity is always processed by the same consumer. That partitioning is the search-space trick: instead of querying a central database for one key among hundreds of billions, each worker answers membership only for its own slice, orders of magnitude narrower. The dedupe worker is a Go program reading its input partitions, checking each message against a local ledger, and publishing only the new ones to the output topic.

The ledger is an embedded RocksDB database on each worker's local EBS drive — a deliberate replacement for the previous Memcached fleet with its atomic check-and-set commit point, its all-keys-in-RAM cost, and its choice between tolerating cache failures or doubling spend on hot failover replicas. RocksDB's LSM shape fits the workload's three simultaneous query patterns: existence checks for random keys that mostly don't exist (the textbook bloom-filter case — most messages are new, and the filter's 'definitely not in the set' answers skip the SSTables entirely), high-throughput appends of new keys (write-ahead log plus sorted memtable, flushed as immutable SSTables, merged by out-of-band compaction), and aging out old keys. Reads and writes are batched — MultiGet against RocksDB, batched consumption from Kafka — trading random access for sequential I/O.

Aging is where the design shows its operational spine: deletion is size-bound, not time-bound. Rather than a fixed TTL, each worker caps its database size and deletes oldest-first through a secondary index of sequence numbers — so a load spike shrinks the de-duplication window instead of toppling the system, and if the window drops under 24 hours, the on-call engineer is paged. The guarantee bends, measurably and monitored, before the system breaks — the explicit inverse of the old Memcached behavior, where mass TTL evictions spiked CPU and exhausted memory.

Correctness closes the loop, and the post is honest that no atomic step spans the three acts — writing RocksDB, publishing to the output topic, acknowledging the input. A crash can land between any two. The resolution is to crown one system: the output topic is the source of truth, doing double duty as write-ahead log, with RocksDB checkpointing and verifying against it. Messages aren't committed from the input topic until RocksDB has persisted them; on restart after a crash, the worker consults the output topic and repairs whichever side diverged. EBS snapshots guard the ledger against corruption, and cycling a worker is a pause, detach, and re-attach that preserves the partition ID. Three months in production: 1.5 TB of keys on disk, roughly 60 billion keys held, 200 billion messages through the system, at 100× the old system's tracking capacity.