Real production systems, taken apart — the dissections and the walls, side by side. https://www.behindscale.com/ 2026-07-27T00:00:00Z https://www.behindscale.com/articles/google-colossus-ssd-placement 2026-07-27T00:00:00Z

Google's own line states the problem: it's hard to read at 50 TB/s if all of your data sits on slow spinning disks — and an SSD-only fleet still carries a substantial cost premium, so the bulk of Colossus's exabytes must stay on HDD while only the data that earns it gets flash. The wall is that placement is decided at the worst possible moment: when a file is created, Colossus can see only the application creating it and its name — the access pattern that would justify the choice doesn't exist yet. The answer is L4, a distributed SSD layer that works as both a read cache and a writeback cache: applications pass features about each new file, L4 groups files into categories, watches each category's real I/O, and continuously simulates competing placement policies — place on SSD for an hour, for two hours, not at all — letting the winning policy govern each category. New files from the right categories are born on SSD and migrated to HDD later; in the best case they're deleted before migration and never touch a spinning disk at all. The same simulations double as a capacity oracle that tells Google how much SSD to buy.

https://www.behindscale.com/articles/linkedin-hodor-overload-protection 2026-07-27T00:00:00Z

Hodor is LinkedIn's overload protection. It runs inside more than 1,000 of their Java services and must never make things worse for members - LinkedIn's logged-in users. Its first version capped how many requests a service handled at once and dropped the rest blindly - a member loading their feed dropped as easily as an offline bulk read. Because members matter more, the fix was to add priority: each request is given a priority - optional, degradable, or non-degradable - and the lowest is dropped first. To act early, each service watches itself for signs of overload: a thread starved of CPU, too much time lost to garbage collection, and requests waiting too long for a free worker. Adding priority then broke how the shedder measured load. It had judged load by counting the requests it was handling, but a dropped request is never handled, so that count can't show the low-priority traffic the shedder must see. It now counts how many requests of each priority arrive instead. Hodor has prevented hundreds of overloads.

https://www.behindscale.com/articles/reddit-piday-outage 2026-07-25T00:00:00Z

Reddit was down for 314 minutes on Pi Day 2023 (March 14), and the cause was configuration that existed nowhere. It was set up years earlier by a team that no longer existed, manually edited through a third-party tool, saved in no repository, and keyed to a label on the servers that a later Kubernetes upgrade deleted. Two minutes into a routine upgrade of the cluster running Old Reddit, the company's most critical legacy system, the site stopped. Responders were nearly blind: the cluster's own metrics died with it, and only the logs survived. Kubernetes has no way to downgrade, so rolling back meant the thing everyone fears: restoring it from a backup, using a guide written long ago for different software, never tried in production, and rewritten on the fly. The postmortem points past the label bug to the real cause: a fleet of clusters each built by hand and all a little different. The fix is a discipline: standardize everything and document it, so no critical setup ever again lives only in a running system and people's memories.

https://www.behindscale.com/articles/slack-incident-2-22-22 2026-07-25T00:00:00Z

On February 22, 2022, Slack was down for hours: many people could not connect. Slack's team stopped the maintenance that had started the outage, and it changed nothing, because by then the system was breaking itself. The maintenance was a routine batched upgrade; each restart briefly dropped cache servers, replaced automatically with empty ones. Two batches had gone fine the week before, but the third ran at peak traffic, when enough of the cache was empty to expose an expensive query it normally hid. One common startup request, looking up who is in your group chats, was cheap only while cached. With the cache empty, it had to search the entire database at once. The overwhelmed database timed out those very requests, so the cache stayed empty and fed itself. Engineers, not the trigger, ended it. They capped how many apps could start at once, protecting connected users and letting the cache refill; when they raised that cap too far, the overload returned, so they climbed back more slowly. Finally they fixed the expensive request.

https://www.behindscale.com/articles/google-colossus 2026-07-25T00:00:00Z

Nearly everything Google serves (YouTube, Gmail, Drive, Search, and every Google Cloud storage product) runs on one file system, Colossus. It exists because its predecessor hit a wall. The old Google File System (GFS) kept a bookkeeping layer, called metadata, that tracked what files exist and where their pieces live. That layer stopped scaling when Google pushed it to handle Search. Colossus rebuilt the metadata to scale out: parallel servers called Curators handle operations like creating a file, and they store the metadata in Bigtable, a database built to grow. The post credits that one change with scaling Colossus more than 100x past the largest GFS clusters. Around that core sits a smart client library, file servers that stream data straight to applications, and background workers (Custodians) that keep storage healthy. One cluster reaches exabytes across tens of thousands of machines: a shared pool where live serving and batch jobs fill each other's idle gaps, with the most-used data kept on a little fast flash storage.

https://www.behindscale.com/articles/canva-media-dynamodb 2026-07-24T00:00:00Z

Canva's media service stores the identity, ownership, status, and content metadata for every piece of media. It is read-heavy, with most reads for recently created media. For years it was a thin layer over MySQL on Amazon's managed database service (RDS), scaled up, with slightly-stale replica reads. Then its largest tables hit every ceiling at once: schema changes that took six weeks, replication and storage size limits, restart downtime, and a 2TB cap on table files. With the number of media nearing a billion in mid-2017 and doubling, Canva bought time with a few temporary fixes (JSON metadata, denormalizing, ID-range sharding) while migrating live to DynamoDB. For each change, a tiny 'this media changed' message let a worker copy that media's latest version from MySQL into DynamoDB. Hot data moved first, results were compared against MySQL in production, and the cutover ran from a rehearsed checklist with zero downtime, and it costs less to run than the RDS it replaced. Today: 25+ billion media, 50 million more daily.

https://www.behindscale.com/articles/gitlab-database-decomposition 2026-07-24T00:00:00Z

GitLab.com ran on one big Postgres database holding almost all user data (git data aside). For years they scaled it by adding read-only copies and pooling database connections. That bought room but never touched the real limit: every write still had to go to one main server, and that server was already the largest machine available. The long-term fix was to spread the data across many servers by customer account, but the app had never kept each account's data cleanly separate, so that was a huge job. So the team took a reachable step first: split the database in two. Measurement chose where to cut: the CI tables carried about half the writes, so moving them out roughly halved the load. Preparing the split took about a year. The final switch-over was kept small by making the app talk to two databases well before the data was actually split. They designed a near-zero-downtime switch, then chose a planned two-hour outage instead, because it was far easier to undo. After seven full practice runs, the real switch took 93 minutes.

https://www.behindscale.com/articles/meta-silent-data-corruption 2026-07-24T00:00:00Z

A silent data corruption is a hardware error that nothing notices: a faulty chip computes a wrong answer and reports no error, writes no log line, and trips no alert. The bad result flows downstream, spreads across services, and surfaces weeks later as an application bug, with no trail back to the chip that caused it. It is also hard to catch at the source. A chip gets only a few hours of testing at the manufacturer and maybe a couple of days at the company that assembles it into a server, then runs for years, where its flaws worsen with age and heat. Harish Dixit's post describes Meta's answer, drawn from 3+ years in production: since no ordinary signal reveals the failure, you must manufacture the evidence. You ask each machine questions whose answers you already know, and treat a wrong answer as the only symptom you will ever get. Meta does this at two depths: a deep test while a machine is down for maintenance, and a shallow test that runs constantly, right beside the live work. Its central finding is that a fleet needs both.

https://www.behindscale.com/articles/doordash-rabbitmq-kafka 2026-07-24T00:00:00Z

In mid-2019, the Celery-and-RabbitMQ system running over 900 asynchronous tasks at DoorDash kept going down under load - and when task processing went down, DoorDash effectively went down. Sudden bursts slowed RabbitMQ, and its own defenses made things worse. Flow Control throttled the app servers sending tasks, which felt it as network latency, and that piled requests up upstream. harakiri, a setting that kills any worker running too long, then killed the slow ones, whose restarts flooded the broker with new connections and more load. The high-availability mode meant to help instead cut throughput, its failovers taking 20-plus minutes and losing messages. The escape was a custom Kafka-based system: each task's name and arguments on Kafka, behind a wrapper routing tasks to the old or new system by a feature flag. It shipped in two weeks and cut RabbitMQ load 80% within a week. Then the honest part: the new system had its own new problems - one slow message could stall a whole partition, fixed with a small local queue.

https://www.behindscale.com/articles/aws-load-shedding 2026-07-24T00:00:00Z

A field guide to load shedding: deliberately rejecting some requests so a server can finish the rest instead of collapsing under all of them. AWS's David Yanacek opens with a confession. His team spent years hunting the right default for max connections, the setting meant to cap how much work a server takes on, and concluded it can't be made right: too low rejects work the server could have handled, too high lets it drown, and any value that fits today is wrong once traffic shifts. What replaced the knob is a set of rules for overload, which left alone feeds itself: a slow server misses deadlines, clients retry, and the retries pile on more load. The fix is to shed the excess and keep goodput steady, where goodput means the requests served fast enough to be useful, not every request sent. The hard part is choosing what to keep, and that ranking is the article's real subject: what an overloaded server protects first, and why dropping the wrong thing (starting with the load balancer's health check) makes the overload worse.

https://www.behindscale.com/articles/slack-vitess-datastores 2026-07-23T00:00:00Z

Slack stored all its data in MySQL, split across shards by workspace (one team's Slack): every team's data lived on one shard, thousands of teams per shard, with the main application handling all the routing. The model was easy to understand and easy to grow, just add more shards. Then came the question the post puts in bold: what if one team's data doesn't fit the largest shard? By fall 2016 it didn't. The biggest customers' shards had reached the largest hardware money could buy, a few shards ran hot beside a mostly idle long tail, and the scheme could not spread one big customer's load. The fix took three years. Slack moved to Vitess, a system that shards MySQL. It migrated feature by feature, double-writing to both the old and new stores and comparing the results before each cutover. By the end, 99% of its traffic (2.3 million queries per second at peak) ran sharded by finer keys like channel id (a single Slack channel), and the application no longer knew where anything lived.

https://www.behindscale.com/articles/segment-exactly-once-delivery 2026-07-23T00:00:00Z

The one rule of a data pipeline is that it must never lose data, so every layer guarantees at-least-once delivery: retry until a message is acknowledged. But Segment's public API sits at the one boundary where retries can't be made clean: mobile clients. A phone enters a tunnel mid-upload, the events land, the response dies, and the client re-sends what the server already received - measured at 0.6% of all events in a four-week window, enough to swing an e-commerce customer between profit and loss. Amir Abu Shareb's post describes the dedupe system built in three months: every message carries a client-generated unique ID, Kafka routes each ID to the same worker, and an embedded per-worker database (RocksDB) answers 'seen this before?' in memory. Under load the dedup window shrinks instead of the system falling over, and after a crash workers repair their records against the output stream, the source of truth. The result: 200 billion messages through, 60 billion keys held, 100 times the old capacity at a fraction of the cost.

https://www.behindscale.com/problems/ambiguous-timeouts 2026-07-23T00:00:00Z

The request timed out - did the charge happen? Five production answers to the double-payment problem: idempotency keys, where the key's memory lives, replica lag, retry windows, and what still breaks. Build your own and get attacked with their incidents.

https://www.behindscale.com/articles/datadog-incident-response-observer-fate 2026-07-22T00:00:00Z

On March 8, 2023 at 06:00 UTC, a routine security update rolled out automatically across Datadog's servers and, as a side effect, wiped out the network settings on each machine as it restarted. Within an hour, tens of thousands of machines across five regions and three cloud providers dropped off the network at once. For a monitoring company the outage had a cruel twist: not only did customers' monitors go silent, but Datadog's own monitoring, which runs on the same platform, went down with it. Detection came in three minutes from a simple watcher that runs completely outside Datadog's own systems, assumes nothing about how the platform is built, and monitors it from the outside the way any customer would. What follows is the story of the largest incident response in the company's history: several hundred engineers, judgment over rigid checklists, working in shifts through a recovery of about 13 hours. Running through it is one lesson: regions that share no infrastructure can still fail together.

https://www.behindscale.com/articles/uber-kafka-consumer-proxy 2026-07-22T00:00:00Z

Uber runs one of the world's largest Kafka deployments (trillions of messages, petabytes daily), and more than 300 of its microservices use Kafka not for streaming but as a message queue, a job Kafka's core semantics quietly resist. Kafka allows only one consumer to work on a partition, and the safe pattern is to finish one message before starting the next, so each partition works like a single checkout lane: one worker, handling one message at a time, in order. When each job is a slow one-second call to a payment provider, reaching a thousand jobs per second means renting a thousand partitions, each nearly idle. And a single lane means one slow or stuck message holds up every healthy message behind it. Consumer Proxy is Uber's answer, built on top of Kafka rather than replacing it: a proxy that takes over talking to Kafka, hands each message to consumer services individually over gRPC, keeps its own record of which messages finished, and reports progress to Kafka the only way Kafka can record it: "all messages up to here are done."

https://www.behindscale.com/articles/aws-idempotent-apis 2026-07-22T00:00:00Z

Launching one EC2 instance takes several calls under the hood (placement, storage, networking, provisioning the VM), and when one fails transiently, the simplest cure is the best one: retry until it succeeds, a pattern so effective AWS bakes default retries into its SDKs. Malcolm Featonby's Builders' Library piece is about the assumption that cure rests on: that retrying a call causes no extra side effects. The dilemma: a customer running a singleton workload (at most one instance, ever) sees its RunInstances call time out, and retrying could launch a second instance. AWS's answer is an idempotent API contract, where idempotent means a call can be repeated with no extra effect: the caller attaches a unique request identifier (EC2's ClientToken), the service records it with every change as one atomic (all-or-nothing) unit, and for that identifier's lifetime it answers any repeat by replaying the original result instead of redoing the work. A repeat with different parameters is rejected, since a changed request means a changed intent.

https://www.behindscale.com/articles/shopify-pods-architecture 2026-07-22T00:00:00Z

In 2015 Shopify ran out of bigger database servers to buy, so it split its database across many servers (sharding) and kept growing. That solved capacity but quietly created another failure mode. Shopify's developers had a one-line command, Sharding.with_each_shard, written in many places: whenever a platform-wide task ran, it made that task touch every shard in turn. So if even one shard was down, that whole task failed for the entire platform, and every shard added was one more way to bring everything down. In 2016 Shopify reorganized the platform around a new unit called a pod. A pod is a set of shops on their own separate databases, shared with no other pod. Shared workers and servers may talk to only one pod per task, and every request and background job is assigned to exactly one pod, so serving a request needs just one pod online. Sorting Hat sends each request to its pod, and a tool called Pod Mover can move a whole pod to a paired backup data center in about a minute without dropping requests or jobs.

https://www.behindscale.com/articles/segment-centrifuge-database-queue 2026-07-19T00:00:00Z

Segment forwards hundreds of thousands of analytics events per second to hundreds of third-party endpoints like Google Analytics, Salesforce, and webhooks. At any moment dozens of them are failing, with latency spikes, bursts of server errors, and per-customer rate limits. The post retraces the queue designs Segment tried, to show why the obvious buffer keeps failing at its job. A single shared queue lets one slow endpoint block every message behind it. With 200+ endpoints at 99.9% uptime each, that is an hour-long pipeline outage every day. Giving each endpoint its own queue only moves the problem: inside that queue, one whale customer's 50,000 back-to-back messages still block everyone else. True isolation needs a separate queue for every source-and-destination pair - 88,000 of them and growing - more than any system can run affordably. Centrifuge is the replacement: each job is an unchangeable row in MySQL, one owner process (a Director) per database, and delivery order comes from a query instead of the slot a message sits in.

https://www.behindscale.com/articles/stripe-rate-limiters 2026-07-19T00:00:00Z

Stripe runs four admission-control layers in production, with a clear division of labor. Two are rate limiters that shape each user's pace day to day: a token-bucket request limiter (each user draws from a steadily refilling allowance of requests) that rejects millions a month, and a concurrent-requests limiter that caps how many of a user's requests can be in flight at once. The other two are load shedders. They fire only during incidents, and instead of looking at any one user, they look at the whole system and decide which traffic gets the capacity that is left. The shedders encode a criticality ladder: critical methods like creating a charge are protected by a permanently reserved slice of the fleet, and when workers back up, a four-tier drop order (test mode first, critical last) sheds from the bottom and restores slowly, so the system doesn't thrash between dropping traffic and letting it back in. The whole design serves one goal, in the post's words: keep the core of your business working while the rest is on fire.

https://www.behindscale.com/articles/pinterest-sharding-mysql 2026-07-19T00:00:00Z

By September 2011, Pinterest was growing faster than any startup before it, and every piece of its infrastructure was over capacity. The NoSQL systems it had bet on for automatic scaling all eventually broke catastrophically, and a large fleet of MySQL read replicas caused bugs whenever they lagged behind. The rebuild launched in early 2012 and was still the core store three and a half years later. It is hand-sharding in its clearest form: the data is split into 4,096 small databases spread across pairs of machines, with a shared config (in ZooKeeper) recording which machine holds which shards. Every object also gets a 64-bit ID that carries its own address: 16 bits of shard, 10 of type, 36 of row. Joins moved up into the application, schemas moved into JSON, production reads never touch a standby copy, and capacity grows by moving some of a busy machine's shards onto a new machine. Finding any object became a matter of simple math.

https://www.behindscale.com/articles/doordash-aperture-global-failure-mitigation 2026-07-15T00:00:00Z

DoorDash catalogs the failure classes that actually take microservice platforms down — cascading failures, retry storms, death spirals, and metastable failures that will not recover on their own — and audits its own defenses against them: adaptive load shedding, gRPC circuit breakers, autoscaling. The verdict is structural, not operational: every mechanism measures local signals, applies general heuristics, and takes local actions, while every failure class propagates through interactions between services. The post's answer is to evaluate Aperture, an open-source reliability controller that watches the whole system and coordinates mitigation across services — and its honesty is that DoorDash has so far validated Aperture only as a very good load shedder in a test environment, while the global coordination that motivates the argument remains untried.

https://www.behindscale.com/articles/airbnb-orpheus-idempotent-payments 2026-07-15T00:00:00Z

Airbnb's move to a service-oriented architecture (breaking one big application into many small independent services) turned every payment into a distributed transaction: one API call fanning out into downstream calls, each changing state, each able to fail or time out mid-flight. A client that never receives a response can't know whether money moved, and in payments the naive recovery, retrying, is exactly what can charge a guest twice. Airbnb's answer is Orpheus, a general-purpose idempotency library (idempotent meaning a call repeated any number of times has the same effect as calling it once) embedded in each payments service. It rests on four spare ideas: an idempotency key identifying each logical request, request state read and written only on a sharded master database, every non-network step wrapped in a single database transaction, and every error classified as retryable or non-retryable. The claimed result: 99.999% payment consistency while annual volume doubled.

https://www.behindscale.com/articles/netflix-conductor-microservices-orchestrator 2026-07-15T00:00:00Z

Netflix's Content Platform Engineering team runs the processes that get a title ready to stream (studio integration, content ingestion, encoding, deployment to the CDN) as async task flows spanning several days across microservices. Historically these were wired together case by case: pub/sub messages, direct REST calls, and databases each holding a fragment of the state. Netflix sums up the problem in one sentence: there was almost no way to systematically answer 'what is remaining for a movie's setup to be complete?' Conductor is the answer as architecture: a central engine where a JSON blueprint defines the flow, a Decider (a state machine) computes what comes next from the blueprint plus the current state, and every workflow can be tracked, paused, resumed, and restarted. Open-sourced after nearly a year in production, it had run 2.6 million process flows, from simple linear jobs to dynamic workflows that ran over multiple days.

https://www.behindscale.com/articles/cloudflare-byzantine-failure 2026-07-14T00:00:00Z

Cloudflare's control plane is the part that configures and manages the service (its API and dashboard), not the network that carries customer traffic. On November 2, 2020, it was degraded for six hours and 33 minutes, with API success dipping to 75% and the dashboard up to 80 times slower. The trigger lasted only six minutes. One network switch entered a half-working state: healthy enough that failover never fired, but broken enough that one node of etcd lost contact with its leader while every other path stayed up. etcd is the shared store the control plane uses to keep its coordination data in agreement. Three etcd nodes then held three views of reality, tied elections made the cluster read-only, and the database manager read the silence as a dead primary and automatically promoted a standby copy. A known defect then forced a rebuild of every standby, leaving a read-heavy login database on a single, overwhelmed primary for hours. Every layer had a backup; every layer's backup assumed a component is either fully up or fully down.

https://www.behindscale.com/articles/aws-shuffle-sharding 2026-07-14T00:00:00Z

Shuffle sharding is a way to limit how much of a service one bad customer can break, without buying extra hardware. This Builders' Library piece, by Colm MacCárthaigh, makes that blast radius something you configure. Start with eight worker machines where any machine can handle any request: efficient, but every customer shares all eight, so one customer's bad traffic can break everything. Splitting the eight into four fixed groups of two caps the damage at a quarter of the service, but everyone in the hit group goes fully down. Shuffle sharding instead gives each customer its own random pair of the eight machines. There are 28 possible pairs from eight machines, so one bad customer now affects only about 1/28th of the others, and because any two customers share at most one machine, a customer whose software just resends failed requests barely notices. Amazon's DNS service, Route 53, runs this at full scale: 2,048 machines, four to each customer domain (like example.com), giving 730 billion possible combinations.

https://www.behindscale.com/articles/roblox-return-to-service 2026-07-13T00:00:00Z

Starting October 28, 2021, Roblox (fifty million daily players on 18,000 self-managed servers and 170,000 containers) went fully down for 73 hours. What turned deep bugs into three days was structural. Everything ran through one Consul cluster, the coordination layer the whole platform leaned on to find services, check their health, hold locks, and store configuration. The monitoring that would have exposed the problem ran on that same cluster, so responders worked four wrong theories nearly blind. The bugs themselves were two mostly unrelated faults buried deep inside Consul: a new streaming feature that jammed under heavy reads and writes at the same time, and a storage defect that quietly turned 16kB disk writes into 7.8MB ones. Recovery became its own engineering project. It took a state reset, turning streaming off, blocking slow leaders from being elected, rebuilding from cold a cache tier that normally serves a billion requests a second, and letting players back in 10% at a time by steering DNS.

https://www.behindscale.com/articles/slack-scaling-job-queue 2026-07-13T00:00:00Z

Slack's job queue seized in production and stayed seized even after the root cause was fixed: a full Redis needed a little free memory to dequeue, so the fuller the queue got, the less it could drain. That queue runs everything too slow to do inside a web request - every message post, push notification, link unfurl, and billing calculation, 1.4 billion jobs on a busy day at 33,000 per second - and the outage began when database contention slowed job execution and Redis filled to its memory limit. The redesign made the smallest change that removed the failure mode rather than a ground-up rewrite: put Kafka in front of Redis as a durable buffer. A stateless Go gateway (Kafkagate) writes each job to Kafka the moment it arrives, and a relay (JQRelay) feeds jobs into Redis only as fast as workers can drain them, so backlog now lands on disk instead of in the memory the drain depends on. It was rolled out carefully: double-writes, job counts checked at every hop, and heartbeat jobs sent through every partition.

https://www.behindscale.com/articles/airbnb-partitioning-main-database 2026-07-13T00:00:00Z

Heading into a summer where peak traffic grew 3.5x a year, Airbnb's main database was running out of road. It was the original monolith's database, still holding much of the core data, and one feature dominated it: the message inbox drove a third of all writes, growing linearly with traffic. A mid-project discovery sharpened the deadline: on Amazon's hosted MySQL (RDS), the busier the database got, the more its own daily backups threatened to take the whole site down. The classic fix, moving a feature's tables onto their own database, normally costs weeks of migration code and consistency testing. Airbnb instead let MySQL's built-in replication do the hard part. They created a live copy of the database and removed every query that joins inbox tables to others. Then they sent writes to the copy before it was in charge, so the writes failed on purpose, waited for the copy to catch up, and promoted it. Two weeks end to end, seven and a half minutes of message downtime, zero migration code, and a third of the main database's writes gone.

https://www.behindscale.com/articles/notion-sharding-postgres 2026-07-09T00:00:00Z

In Notion, every piece of content (a paragraph, heading, or image) is a 'block', and all of them live as rows in one giant Postgres table. That block table carried five years and four orders of magnitude of growth before its volume began defeating the single database beneath it: VACUUM (Postgres's housekeeping that reclaims space from dead rows) stalled, and behind it waited transaction-ID wraparound, a safety mechanism that halts all writes and would be fatal for a write-heavy product. The answer was application-level sharding, chosen over off-the-shelf tools like Citus and Vitess so Notion could control its data placement: 480 logical shards (picked for its factors) across 32 physical databases, partitioned by workspace ID, with related tables kept together so transactions never cross hosts. The migration ran four phases: double-write via an audit log, a three-day backfill on 96 CPUs, sampled verification and dark reads written by different people than the migration, and a five-minute switchover the team says could have been zero.

https://www.behindscale.com/articles/meta-foqs-priority-queue 2026-07-09T00:00:00Z

Facebook Ordered Queueing Service (FOQS) is the queue underneath Meta's asynchronous world, processing close to a trillion items a day for hundreds of use cases, from notifications to video encoding. Its defining promise is what happens when consumers fail: enqueues keep landing, backlogs grow into the hundreds of billions of items, and nothing is lost - downstream failure becomes delay, not loss. Keeping that promise is the hard part, because FOQS is built on sharded MySQL, and MySQL slows down at exactly the wrong moment. The more pending work piles up, the more old row versions the database must drag through on every scan. The article walks the engineering that keeps a hundred-billion-item backlog from slowing the queue that holds it. Four moves do the work. Checkpoints keep each scan from re-reading old history. Per-shard indexes are merged into one priority order. Short delivery leases (a countdown a consumer must confirm within) turn a crash into a redelivery, not a loss. And circuit breakers stop feeding unhealthy shards.

https://www.behindscale.com/articles/figma-postgres-sharding 2026-07-06T00:00:00Z

Figma's database stack grew almost 100x since 2020, and by late 2022 its largest tables were each nearing the limits of a single database server. Vertical partitioning can't fix that, since the smallest thing it can move is one whole table. Rather than switching to a NewSQL or NoSQL database, the databases team split those tables across many servers from inside their own application. The design: a small Go proxy (DBProxy) with a minimal query engine, a few chosen shard keys, hashed so data spreads evenly, and related tables grouped into 'colos' so common joins and transactions keep working. The signature move was to make a table act sharded before moving any data: Postgres 'views' plus on/off flags let the team rehearse the whole sharded setup on the original server, with rollback in seconds. Only after that rehearsal proved itself in production did they run the real, hard-to-reverse split. The first table shipped in September 2023, roughly 9 months in, with about 10 seconds where some writes failed on the main databases.

https://www.behindscale.com/articles/slack-cellular-architecture 2026-07-06T00:00:00Z

On June 30, 2021, a flaky network link in one AWS availability zone degraded Slack for everyone. The incident review's real question was why a failure in just one zone reached users at all. The answer was a gray failure: different parts of the system disagreed about what was actually up. A single user request fans out into hundreds of internal calls that all have to succeed. And Slack's main database keeps each piece of data on one machine that must be reachable to save changes, so when the flaky link cut those machines off, those saves failed. A partial, confusing failure in one zone turned into errors everywhere. Rather than try to detect these gray failures automatically, Slack spent 1.5 years moving its most important user-facing services to cells. Every service runs in every zone but talks only to services in the same zone, turning each zone into a self-contained cell. The result is a single control that pulls all traffic out of a troubled zone in seconds, 1% at a time, without needing anything inside that zone to still work.

https://www.behindscale.com/articles/uber-cadence-workflow-platform 2026-07-06T00:00:00Z

Cadence is Uber's answer to a question every large company faces: where should the durable execution of async workflows live? Uber built one central workflow platform and made it serve over a thousand services, from the most critical (which Uber calls T0) down to the least (T5). It runs 12 billion executions and 270 billion actions a month at 99.9% availability, with workflows written as ordinary code in normal programming languages rather than in a special configuration format. The bet is that the engine is built once while every workflow is written fresh, so the hard parts belong in the central workflow platform, not in each team's code. Uber's 2021 internal survey found teams writing 40% less code for the same functionality. The costs are the platform's to carry. A team of 20 runs a shared dependency the whole company leans on. Keeping one heavy team from slowing down the others is constant work. And the shared database underneath caps how much the whole platform can handle.

https://www.behindscale.com/articles/shopify-resilient-payments 2026-07-06T00:00:00Z

A Staff developer on Shopify's payments team distills five years of running payment infrastructure into ten practices. Bound every wait (timeouts, circuit breakers, admission control), make every retry safe (idempotency keys with recovery steps), and verify the money afterward (reconciliation with recorded anomalies). All of it is wrapped in a loop of golden-signal monitoring, correlated logs, flash-sale load tests, and a disciplined incident process. The post has no single architecture to unveil. Its value is the dependency chain between small mechanisms, each one bounding a failure mode the previous one lets through.

https://www.behindscale.com/articles/github-partitioning-relational-databases 2026-07-06T00:00:00Z

For over a decade, GitHub.com revolved around one MySQL cluster, mysql1, holding users, repositories, issues, and pull requests. Between 2019 and 2021, GitHub partitioned it without downtime, and the order was the point: first in the code, then in the database. In the code, they grouped related tables into named schema domains and added linters that made any query crossing a domain boundary fail the build, so the app behaved as if split before a single byte moved. Only then came the database move: whole tables shifted to new clusters with Vitess (the MySQL scaling layer born at YouTube) and a custom write-cutover whose read-only window lasted tens of milliseconds. They moved 130 of their busiest tables in one cutover. The result: the same data now answers 1.2M queries/s across several clusters, up from 950k, while average per-host load halved.

https://www.behindscale.com/articles/aws-timeouts-retries-backoff-jitter 2026-07-06T00:00:00Z

Marc Brooker's Builders' Library article is Amazon's operational doctrine for the three tools every remote call needs: timeouts so failures surface instead of hanging, retries so transient faults get masked, and backoff with jitter so the masking doesn't become the next outage. Its sharpest idea is that retries are selfish — a client retrying spends more of the server's capacity to improve its own odds, which amplifies load precisely when the dependency is least able to absorb it. Amazon's answers: retry only when the dependency looks healthy and stop when retries aren't improving availability; throttle retries client-side with a token bucket (built into the AWS SDK since 2016); treat side-effecting APIs as unsafe to retry unless they're idempotent, the way EC2 RunInstances is via client tokens; and add jitter not just to retries but to every timer, periodic job, and piece of delayed work, because correlated schedules manufacture their own load spikes.

https://www.behindscale.com/articles/netflix-prioritized-load-shedding 2026-06-13T00:00:00Z

Netflix pushed prioritized load shedding down from the API gateway into individual services, letting a single service protect its most critical traffic without splitting it onto separate clusters. Inside PlayAPI, a partitioned concurrency limiter (a cap on how many requests are processed at once, divided by priority) guarantees user-initiated playback requests full throughput while lower-priority pre-fetch requests get only leftover capacity: the isolation of separate clusters at the compute cost of one. A production incident proved it out: after an outage, a 12x pre-fetch spike was shed down to 20% availability while user-initiated playback held above 99.4%.

https://www.behindscale.com/articles/discord-trillions-message-search 2026-06-09T00:00:00Z

Discord redesigned its message search to handle trillions of messages, replacing a 2017 system that had served well for eight years but had reached its limits. The redesign keeps the one decision that aged well, routing each message to its shard in application code, while replacing everything that routing hands the message to next. The Redis queue becomes Google Cloud PubSub, which never drops messages under load. The two large Elasticsearch clusters become about forty smaller ones grouped into logical 'cells'. Adding messages to the search index in bulk (many at once) learns to group each batch by where it is going. And the few guilds (Discord servers) nearing two billion documents, the hard limit of Lucene (the search engine underneath Elasticsearch), get their own cells with indices split across several shards.

https://www.behindscale.com/articles/airbnb-monitoring-reliably-at-scale 2026-06-02T00:00:00Z

Airbnb's monitoring system ran on the same shared infrastructure it was supposed to watch, so when that infrastructure broke, the monitoring broke with it, going blind at the exact moment engineers needed it most. This is a circular dependency: the tool meant to detect an outage depends on the very thing that is failing. Airbnb rebuilt its monitoring to remove that loop, in three layers. First, it moved the monitoring onto its own dedicated machines, still run by the platform team but not shared with the services being watched. Second, it gave monitoring data its own network path, separate from the shared networking layer that every other service uses. Third, it added a 'dead man's switch': the monitoring sends out a steady heartbeat, and if that heartbeat ever stops, the silence itself is what pages the on-call engineer. The one rule running through all of it: never let your safety net depend on the thing it is supposed to catch.

https://www.behindscale.com/articles/uber-intelligent-load-management 2026-06-02T00:00:00Z

Uber's load management for its distributed databases (Docstore and Schemaless) evolved through three phases. Quota-based rate limiting failed because it ran too far from the storage nodes, up in the stateless query layer where overload doesn't actually show up. CoDel, a queue-trimming algorithm borrowed from networking that sheds requests by how long they have waited, stabilized the storage engine but treated a ride request and a batch job as equals. Cinnamon, the current generation, adds the judgment: it sheds by priority tier (t0, most critical, down to t5), continuously nudges its rejection threshold up and down against measured overload using a PID controller (a self-correcting loop, explained below), and accepts any overload symptom as a plug-in signal through a bring-your-own-signal design (BYOS). The result under overload: about 80% more throughput, roughly 70% lower P99 latency, and far fewer goroutines (Go's lightweight threads) piling up during overload events.

https://www.behindscale.com/articles/skipper-workflow-engine 2026-06-02T00:00:00Z

Airbnb built Skipper as a workflow engine that ships as a library embedded inside each service, not as a separate central cluster. Durability comes from replay: when a service restarts, the workflow method simply runs again, skipping every action already saved (checkpointed) to the database. A workflow that has to wait uses no compute at all while it waits. If it fails midway, Skipper automatically undoes the steps that already ran, in reverse order. And all of this state lives in the database the service already uses. The whole design is built around one choice: pay almost nothing when things go smoothly, and call on the recovery machinery only when something goes wrong. In production over a year across 15+ use cases, peaking at 10,000 workflows/second.

https://www.behindscale.com/articles/stripe-idempotency 2026-05-29T00:00:00Z

A network call that times out leaves the client unable to tell whether the operation happened. For a payments API, both guesses are catastrophic. Retry a charge that succeeded and you double-charge the customer; fail to retry one that failed and they never get what they paid for. The post lays out how Stripe makes its APIs safe to retry, in three layers. Where HTTP semantics allow it, endpoints are made idempotent - built so that repeating a request produces the same result as sending it once. Where an operation must happen exactly once, the client sends an idempotency key so the server can recognize a retry. And clients retry with discipline - exponential backoff plus jitter - so that a fleet of recovering clients never becomes the next outage.