At Most Once: Orpheus and the Idempotent Payments Library at Airbnb
Airbnb's move to a service-oriented architecture (breaking one big application into many small independent services) turned every payment into a distributed transaction: one API call fanning out into downstream calls, each changing state, each able to fail or time out mid-flight. A client that never receives a response can't know whether money moved, and in payments the naive recovery, retrying, is exactly what can charge a guest twice. Airbnb's answer is Orpheus, a general-purpose idempotency library (idempotent meaning a call repeated any number of times has the same effect as calling it once) embedded in each payments service. It rests on four spare ideas: an idempotency key identifying each logical request, request state read and written only on a sharded master database, every non-network step wrapped in a single database transaction, and every error classified as retryable or non-retryable. The claimed result: 99.999% payment consistency while annual volume doubled.
Charge a guest $100, cut the response after the bank has moved the money, and choose what the retry does — with Orpheus off, on, and on-but-reading-a-replica. The double charge you cause in replica mode is the post's own scenario.
Problem
The requirements elevate this beyond a textbook idempotency post. First, it had to be general: one configurable framework shared across all payments services, not a separate hand-built solution for each use case. Second, uncompromising: payment consistency could not degrade while the migration to many services was in flight, because inconsistency lands directly on guests and hosts. Third, ultra-low latency, which produced the post's sharpest judgment. A standalone idempotency service (a separate service every payment call would consult over the network) was rejected not just for the extra network hop, but because that service would suffer from the same problems it was meant to solve. A guard reached over the network is itself a remote call that can time out ambiguously, so it would inherit the very disease it was built to cure. Fourth, organizational: with the company splitting into many services and the engineering team growing, requiring every product developer to become an eventual-consistency specialist would be highly inefficient, so the machinery had to live below the product code, where an ordinary developer never has to reason about it.
The post is unusually specific about the failure modes to absorb: a client that fails to consume the response, a response lost in transit, a client-side timeout, and a race condition from a user clicking 'Book' twice. Each is the same gap in what the client can know - the work may or may not have happened, and there is no evidence either way - arriving through a different door. And each has to be survived without the one shortcut that would make it easy, trusting the client to know what already happened, because the whole point is that the client cannot know.
Solution
Orpheus rests on a split the post treats as nearly universal: almost every API request breaks into three phases. Pre-RPC, where the payment request's details are recorded in the database; RPC (the remote procedure call, the live request across the network to the external payment processor or bank), where the call goes out and the response comes back; and Post-RPC, where the outcome is recorded, including whether a failure is retryable. Two ground rules fence the phases: no network calls in Pre and Post, no database work in RPC. Network and transactions never mix, a rule learned the hard way through connection-pool exhaustion, where holding a database connection open across a slow network call drained the pool and degraded the whole service. Each of Pre- and Post-RPC is wrapped in a single enclosing database transaction started by the library, so each phase either fully succeeds or fully fails: the system can be interrupted anywhere and still be in a state it can recover from. Java lambdas make the enclosure ergonomic, letting application database work pass as a function into the library's transaction.
Retry safety hangs on exception discipline. Every error is classified retryable or non-retryable: infrastructure failures (5XX-shaped) are presumed temporary and retryable; validation failures (4XX-shaped, like trying to refund a refund) are permanent, and the custom exception class defaults to non-retryable. The post spells out the cost of getting either label wrong: a retryable error wrongly marked non-retryable fails the request forever; a non-retryable one wrongly marked retryable re-opens the door to double payments plus manual cleanup. The client carries real responsibilities: create one unique key per request and send that same key on every retry; save the key before making the call; on success, record it and retire the key; never change the request's contents between tries; and space out automatic retries using longer and slightly randomized waits (so a burst of retries doesn't all land at once). How the key is built is itself a design choice. Request-level keys use a random unique ID for each attempt at a call - good for recognizing a retry of that one call. Entity-level keys are instead built from the thing being acted on, like payment-1234-refund; because that key is the same every time, the framework can guarantee a given refund happens only once ever, even across unrelated requests that never coordinated. The client picks whichever the situation needs. Two attempts on the same key are kept from running at once with a lease: the first call locks its own row in the database, which grants it the sole right to be in flight; that lock is time-limited (set longer than the network call's timeout) so a crashed or hung server doesn't leave the key stuck forever, and a cap on how long retries are allowed stops runaway retrying. Once a request reaches a final outcome, that outcome is saved and simply handed back to any later retry - which costs a table that keeps growing with traffic and is hard to trim or change the shape of later.
Then the post's best section: why the idempotency tables live only on the master database (the authoritative copy that takes all writes), never on a replica (a trailing copy kept for reads). With MySQL and no strong read consistency, storing this state on a replica brings the original problem right back inside the fix: a payment succeeds, the response commits on master, the client's timeout fires a correct idempotent retry, and the retry, reading a replica that hasn't caught up yet, finds no recorded response, concludes the payment never happened, and runs it again. A few seconds of replica lag equals a double charge. So Orpheus reads and writes on master only, and handles the load on that single hot spot by splitting the tables across shards using the idempotency key itself - which works well as a split point because its values are numerous and spread evenly. The claimed outcome: 99.999% consistency while annual payment volume doubled.
Tradeoffs
- Building this as a library instead of a separate service is the right call here, and the reason applies broadly: a separate idempotency service would add the network delay the requirements ruled out, and it would itself be reached over an unreliable network - so the thing meant to prevent ambiguous failures would be exposed to ambiguous failures of its own. The price of the library is that every service now carries the machinery, upgrades have to roll across all the services that adopted it, and the separation between everyday product code and the consistency machinery is kept by agreement and careful API design rather than enforced by a hard network boundary.
- Splitting each request into three phases buys the ability to recover from a mid-way failure, but at a real cost to the developers, which the post lists plainly: API calls have to be refactored into three chunks, which is restrictive, genuinely hard for complex calls, and demands forethought about what crosses each phase boundary. Their own escape hatch shows the strain: one service models its flow as a state machine where every transition is itself an idempotent step, packing many small three-phase units inside a single API call.
- Sorting each error into 'retryable' or 'non-retryable' - the classification the solution relies on - is load-bearing and has to be done by hand. The whole retry guarantee rests on a person judging, for each way a call can fail, whether trying again is safe, and both kinds of mistake are expensive: mark a recoverable error 'permanent' and the request fails forever; mark a permanent one 'retryable' and you get double payments plus manual cleanup. The framework can supply a safe default (non-retryable, the cautious choice for money) but it can't make the judgment go away.
- Saving every final response trades growing storage for fast, correct retries. The table of saved responses grows with traffic; deleting old rows risks breaking a retry that arrives late; and the saved formats can't be changed in any way that would stop old rows from being read back. This store looks like a cache but can't be treated like one - you can't just clear it, because a late retry may still depend on what's in it.
- Master-only reads trade scalability for correctness, then buy the scalability back with sharding. Reading this state from a lagging copy isn't a speed-for-accuracy tradeoff that degrades gently; it's an outright correctness hole, exactly as wide as the copy's lag. Splitting by idempotency key wins the throughput back, at the cost of the payments system's most critical tables now living on a split, write-heavy path.
- Write repair pushes work onto the client by design. Clients persist keys before calling, own their retry policy, and must never change payloads; in exchange they get consistency on demand, firing the same request until the system converges. Putting the smarts in the client and keeping the server simple is the opposite of the usual instinct, and it's exactly what makes the retry safe to fire from total ignorance.
Patterns in this article
- Idempotency Keys
This is the deepest look the library has at what an idempotency key actually does inside a service. Stripe defines the key as an API contract, Shopify shows why it matters at volume, and AWS pairs it with retry discipline; Airbnb shows the interior. Here the key is three things at once: the row a request locks while it runs, the value the tables are sharded on, and a design decision in its own right, since a random UUID per request behaves differently from a deterministic key like payment-1234-refund (which lets a given refund happen only once, ever).
- Retry with Backoff and Jitter
The client's half of the contract. Automatic retries have to be spaced out with growing waits and a little randomness (backoff and jitter) so that a wave of retries doesn't all hit at once and overwhelm the service (the 'thundering herd'). And the retry is only safe to fire at all because the idempotency key guarantees it can't double-charge. It pairs with the AWS article's point from the other side: AWS shows that retries spend the server's capacity, and Airbnb's framework at least makes spending it harmless to correctness.
- Atomic Phases
The same all-or-nothing idea appears here at a smaller scale than in Skipper. Skipper draws its phase boundaries around whole workflow steps (checkpointed actions); Orpheus draws them inside a single request, splitting it into Pre-RPC / RPC / Post-RPC and fencing the phases with two rules: no network calls inside a database transaction, and no database work during the network call. Both give the same property, that you can be interrupted partway and still recover cleanly, just at different sizes: one around a whole workflow step, the other around the parts of one request.
- Retryable Error Classification
Every failure is sorted into two buckets: retryable ones, presumed temporary and safe to try again under the same key, and non-retryable ones, treated as permanent so the recorded failure is simply replayed. The default leans to non-retryable, the safe direction for money. This is what turns 'a retry is safe' from a vague hope into a concrete, per-error rule the framework can act on.
- Master-Only Reads
Airbnb reaches this rule through the sharpest possible failure. If the idempotency tables are read from a replica (a copy that trails slightly behind the authoritative master), a correct retry can turn into a double charge: the payment succeeded and was recorded on master, but the retry reads a replica that hasn't caught up, sees no record, and runs the payment again. A few seconds of lag equals a double charge. So Orpheus reads and writes only on master, and wins back the lost scale by sharding on the idempotency key. Pinterest arrives at the same rule from a different angle: replicas are for surviving disaster, not for serving reads.
Also solving this
Other systems in behindscale's Ambiguous failure under retry class: