When Queues Stop Working: Segment's Database-as-a-Queue

The post's problem statement is a tour of its own failed architectures, each one a lesson priced in production. Architecture one: a single queue, workers popping jobs and calling whatever API each needs. One slow endpoint creates backpressure on the entire message flow — and the arithmetic is merciless: 200+ endpoints, each 99.9% available (an hour of downtime a year), means the pipeline eats an hour-long outage once per day. Architecture two: a router publishing into a dedicated queue per destination, so a failing API only strands its own messages. Better — until the multi-tenant power law shows up inside each queue. Customer A sends 50,000 contiguous messages to an endpoint rate-limited at 1,000 calls per second per customer; customers B and C sit behind them. The post lays out the three exits and indicts each: hard-cap A and delay B and C by 50 seconds; keep sending and burn retries into 429s; or detect the limit and copy A's remaining 49,000 messages to a dead-letter queue — terabytes on the move at Segment's scale, since Kafka's delivery order is effectively set by the producer and reordering means rewriting.

The ideal is obvious and unbuildable: a queue per source-destination pair, with per-pair consumption rates. Segment counts 42,000 active sources sending to an average of 2.1 destinations — 88,000 queues and growing quickly. Across Kafka, RabbitMQ, NSQ, and Kinesis, nothing supports that cardinality with sane scaling; SQS can, at a cost the post calls prohibitive. The requirements crystallize into three: per-customer isolation so one tenant's failing traffic can't slow global delivery; reordering without copying terabytes; and horizontal scaling without perpetual repartitioning. What's needed, the post concludes, is a new primitive.

The primitive is the database-as-a-queue. Centrifuge's unit is a job — a payload plus an endpoint, with headers governing retries, encoding, and timeouts — and every job lives as a row in a MySQL (RDS) instance called a JobDB. Three properties make a database viable where intuition says it's too slow. Rows are immutable: state changes append to a separate job_state_transitions table (awaiting-scheduling → executing → succeeded / discarded / awaiting-retry → archiving → archived), so the database never runs updates. There are no JOINs: every query is per-job, so databases parallelize without coordination. And the workload is write-heavy with a tiny working set: jobs live a few hundred milliseconds, so nearly everything is served from an in-memory cache and expired on delivery — production Cloudwatch shows writes utterly dominating reads. The primary key is a KSUID — k-sortable by timestamp and globally unique — so one index serves both point lookups and time ordering. A 400-class response is terminal (discarded: the server will reject it every time); 500s, timeouts, and disconnects are ephemeral and retried with exponential backoff until a four-hour expiry, after which jobs archive to S3 for out-of-band reprocessing.

The writer architecture is the second inversion. Instead of many stateless workers sharing databases, exactly one Director — a Go process — owns each JobDB, acquiring it through a Consul session that guarantees mutual exclusion. Sole ownership means the Director manages caching, locks, and queries entirely in-process with zero coordination: it logs incoming jobs (accepted via RPC), executes the HTTP deliveries, appends every transition, and only reads from the database when recovering from a failure. Scaling is symmetrical: 80 to 300 Directors run at peak on CPU-based autoscaling, and a JobDB Manager keeps the database fleet matched — holding spares for sudden scale-up, retiring instances off-peak, and cycling every JobDB roughly every 30 minutes so the working set never outgrows RAM. Cycling converts millions of random deletes into a single drop table; a drainer process migrates still-retrying jobs to a fresh database first, since by the 30-minute mark 99.9% of events have already succeeded or failed.

The production numbers carry the argument: nine months, five engineers, 50,000 lines of Go; 400,000 outbound HTTP requests per second (load-tested to 2 million); 340 billion jobs in the reported month. On average 1.5% of all global data succeeds only on a retry — the post's candid framing: almost meaningless to an early-stage startup, incredibly significant to a large retailer. And the system's first real trial reads like the design brief fulfilled: on March 17th a popular integration taking 16,000 requests per second degraded to 15% success for 105 minutes. Centrifuge absorbed roughly 85 million events, retried on backoff — admittedly peaking at 100,000 requests per second against the struggling partner, a tuning gap the post owns — and delivered the entire backlog within 30 minutes of recovery. Mutual customers saw delays, not loss. No other integration noticed.