When Queues Stop Working: Segment's Database-as-a-Queue
Segment forwards hundreds of thousands of analytics events per second to hundreds of third-party endpoints like Google Analytics, Salesforce, and webhooks. At any moment dozens of them are failing, with latency spikes, bursts of server errors, and per-customer rate limits. The post retraces the queue designs Segment tried, to show why the obvious buffer keeps failing at its job. A single shared queue lets one slow endpoint block every message behind it. With 200+ endpoints at 99.9% uptime each, that is an hour-long pipeline outage every day. Giving each endpoint its own queue only moves the problem: inside that queue, one whale customer's 50,000 back-to-back messages still block everyone else. True isolation needs a separate queue for every source-and-destination pair - 88,000 of them and growing - more than any system can run affordably. Centrifuge is the replacement: each job is an unchangeable row in MySQL, one owner process (a Director) per database, and delivery order comes from a query instead of the slot a message sits in.
Run the same two disasters, a partner outage and a whale customer's flood, through all three of Segment's real architectures. On the shared queue, everyone waits behind the failure. On per-destination queues, pick your poison: delay the innocent, retry into rate limits, or copy terabytes. Then switch to Centrifuge and watch the stuck jobs step out of everyone else's way.
Problem
The post's problem statement is a tour of its own failed architectures, each one a lesson priced in production.
- Architecture one is a single shared queue: workers pop jobs and call whatever API each one needs. One slow endpoint backs up the entire flow. With 200+ endpoints, each 99.9% available (about an hour of downtime a year), the pipeline eats an hour-long outage once per day.
- Architecture two gives each endpoint its own queue, so a failing API only strands its own messages. That is better, until a few big customers dominate one endpoint's queue. Customer A sends 50,000 back-to-back messages to an endpoint that allows only 1,000 calls per second per customer, and customers B and C are stuck behind them.
Every exit from that jam is bad. Hard-cap A and you delay B and C by 50 seconds. Keep sending and you burn retries hitting the rate limit again. Or copy A's remaining 49,000 messages to a side queue, which means moving terabytes at Segment's scale, since Kafka's delivery order is fixed by the sender and reordering means rewriting.
The ideal is obvious and unbuildable: a separate queue for every source-and-destination pair, each with its own delivery rate. Segment counts 42,000 active sources sending to an average of 2.1 destinations, which is 88,000 queues and growing quickly. Across Kafka, RabbitMQ, NSQ, and Kinesis, nothing runs that many queues with sane scaling. SQS can, at a cost the post calls prohibitive. So the requirements come down to three:
- per-customer isolation, so one customer's failing traffic cannot slow everyone else's delivery;
- reordering without copying terabytes;
- adding capacity without constantly re-splitting the data across machines.
What is needed, the post concludes, is a new building block.
Solution
The building block is the database-as-a-queue. Centrifuge's unit is a job: a payload plus an endpoint, with headers that govern retries, encoding, and timeouts. Every job lives as a row in a MySQL database (on Amazon RDS) called a JobDB. Three properties make a database fast enough here, where intuition says it is too slow:
- Rows never change. State changes are appended to a separate table (from awaiting-scheduling, to executing, to succeeded, discarded, or retrying, and finally archived), so the database never rewrites a row.
- Every query touches a single job. There are no JOINs, so many databases work in parallel without coordinating.
- The workload is write-heavy with a tiny working set. Jobs live a few hundred milliseconds, so nearly everything is served from an in-memory cache and dropped on delivery. Monitoring shows far more writes than reads.
The primary key is a KSUID, an ID that sorts by time and is globally unique, so one index serves both direct lookups and time ordering. When a delivery fails, the response says whether to retry. A 4xx client error means the request itself is bad, so the job is discarded and never retried. Server errors, timeouts, and dropped connections are treated as temporary and retried with exponential backoff until a four-hour cutoff, after which the job is archived to S3.
Centrifuge inverts the usual setup twice. The first inversion is using a database where you would normally use a queue. The second is ownership: instead of many identical workers all sharing the databases, exactly one process (a Director, written in Go) owns each JobDB. It claims that ownership through a Consul session, a lock that guarantees only one Director holds a given database at a time. Because nothing else can touch its database, the Director keeps all caching, locks, and queries in its own memory, with no coordination. It records incoming jobs, makes the HTTP deliveries, appends each state change, and only reads from the database when it has to recover after a crash.
Scaling works the same way up and down. At peak, 80 to 300 Directors run under CPU-based autoscaling, and a JobDB Manager keeps the number of databases in step with them. It holds a few spares for a sudden spike, retires databases off-peak, and replaces each JobDB every 30 minutes or so, so its working set never outgrows memory. Replacing a database turns millions of small deletes into a single drop table. Just before that, a drainer moves the few still-retrying jobs to a fresh database, since by the 30-minute mark 99.9% of events have already succeeded or failed.
The production numbers make the case: nine months, five engineers, and 50,000 lines of Go; 400,000 outbound HTTP requests per second, load-tested to 2 million; and 340 billion jobs in the reported month. On average, 1.5% of all data succeeds only after a retry. The post is candid about what that means: almost nothing to an early-stage startup, and a great deal to a large retailer. The system's first real test read like the design brief fulfilled. On March 17th, a popular integration taking 16,000 requests per second dropped to 15% success for 105 minutes. Centrifuge soaked up roughly 85 million events and retried them on backoff, then delivered the entire backlog within 30 minutes of recovery. Its retries did briefly peak at 100,000 requests per second against the struggling partner, a tuning gap the post owns. The shared customers saw delays, not lost data. No other integration noticed.
Tradeoffs
- Swapping the queue for a database gives up simple operations in return for control over the data. Changing delivery order is now a SQL statement and a deploy, not terabytes copied across the network. The cost is running databases as throwaway infrastructure: a Manager, a pool of spares, 30-minute replacement, and paired drainer processes. Segment now runs a fleet of MySQL databases the way other teams run stateless containers.
- Letting one Director own each database makes writes fast, because no other process can write and nothing has to be coordinated across machines. The cost is availability at a fine grain: each Director is the single point of failure for its slice of the data, and if it crashes, the replacement has to rebuild its state from the database. The Consul lock that keeps ownership exclusive becomes safety-critical, because if two Directors ever wrote to the same database at once, they would corrupt the very guarantees the design exists to provide.
- Unchangeable rows and 30-minute replacement make deletes cheap and recovery predictable, but they add moving parts. Because rows are only appended, a job's current state has to be reconstructed from its history rather than read from one field. And because whole databases are dropped every 30 minutes, a drainer has to first move out the 0.1% of jobs still retrying, a background job that must never lose one.
- The isolation between customers is not physical. The 88,000 per-pair queues are just a scheduling rule applied to rows, not 88,000 real queues, which is exactly what makes so many of them affordable. But because customers still share a Director and a database, that isolation holds only as long as the scheduling logic is correct. The wall between them is code, not hardware.
- An absorber this good can become the second incident. During the March outage, Centrifuge's retries peaked at 100,000 requests per second against a partner rated for 16,000, which the post admits shows the strategy still needs tuning. A buffer that never gives up has to decide how hard to keep knocking on a door that is not answering. Exponential backoff smoothed the curve, but only after the initial burst.
- Building the block cost what buying could not: nine months, five engineers, 50,000 lines of Go, plus a custom QA and load-testing setup. No existing queue offered the isolation at this many queues, and SQS priced out. The recurring dividend is 1.5% of all data recovered by retry, and outages that stay inside one integration. The post is honest that the same 1.5% would be a rounding error for a smaller company. This is a building block you make at a particular scale, not before.
Patterns in this article
- Database-as-a-Queue
A term the post coined for its central idea: when a buffer needs richer access than just push and pop (per-customer reordering, priority changes, selective retry), store the jobs as unchangeable rows and make delivery order a query. The usual warning that databases make poor queues holds only until the workload fits three shapes: rows that are only appended, queries with no JOINs, and a write-heavy, few-hundred-millisecond working set served from cache.
- Single-Writer Ownership
One Director owns each database exclusively, through a Consul lock, so caching, locking, and cache-invalidation all happen inside one process with no coordination between machines. The writes stay fast because no one else can write. The lesson runs against the usual default of many identical workers over shared databases: when a workload is write-heavy with a small working set, pairing one owner to one database beats sharing.
- Retry with Backoff and Jitter
Another company leaning on this pattern, at industrial scale: retrying is one of Centrifuge's three core jobs, controlled per-job by headers, capped at a four-hour cutoff, and measured. 1.5% of all data succeeds only on a retry, and half of those wins arrive on attempts three through ten. The March outage shows the other edge: the retries peaked at 100,000 requests per second against a partner rated for 16,000 before backoff smoothed the curve.
- Fault Isolation
This is the multi-tenant version of the pattern: one customer's failing traffic must not slow anyone else's delivery. The March outage is the proof: one integration was degraded for 105 minutes while no other integration noticed. The twist is that the walls between customers are virtual, enforced by a scheduling rule over database rows rather than by physically separate queues.
Also solving this
Other systems in behindscale's Buffer degrades under backlog class: