Ten Bounds on Failure: Resilient Payment Systems at Shopify

The post is unusual for an engineering blog: it is not the story of one system, but a Staff developer's distillation of five years on Shopify's payment infrastructure into the ten practices he teaches new team members. Bart de Water writes that what was mostly consistent among engineers joining the payments team — whatever their language or domain background — was little experience building systems at Shopify's scale. The post is the onboarding curriculum for that gap, boiled down from an internal team presentation.

The environment those practices exist for has two defining properties. The first is that payment processing lives at the boundary between Shopify and its financial partners, across networks that are unreliable even when they look reliable most of the time. At Shopify's scale, a once-in-a-million chance of something unreliable occurring during payment processing means it happens many times a day. And in payments, an ambiguous outcome is not an inconvenience: if an API call to a partner times out, the request must be retried — but a careless retry double-charges the card, which is not just annoying for the cardholder but exposes the merchant to a chargeback if the duplicate goes unnoticed, and a double refund harms the merchant directly. De Water frames the stakes with a Dutch saying — trust arrives on foot, but leaves on horseback — and notes that merchants' livelihoods depend on Shopify Payments when they choose it.

The second property is load, and the post grounds it in queueing theory. Little's Law — the average number of customers in a system equals their average arrival rate multiplied by their average time in the system — rearranges into capacity = throughput * latency: 50 requests arriving in a queue at an average of 100 milliseconds each is a throughput of 500 requests per second. When incoming work exceeds capacity, the queue grows until clients waiting on it time out — at which point, from the client's perspective, the service is down. In practice the danger starts well before full utilization: processing time is not uniformly distributed, so queue size starts growing somewhere around the 70 to 80 percent saturation mark, and with enough incoming volume servers can run out of memory holding queued work and crash. Against all of this, the platform defaults are hostile: Ruby's built-in Net::HTTP allows 60 seconds each to open a connection, write data, and read a response, while Go's http.Client and Node's http.request ship with no default timeout at all — an unresponsive server can tie up resources indefinitely.

The ten tips read as a list, but they chain into a coherent defense in depth: bound how long anything can wait, stop calling what is known to be down, bound how much work is admitted, make the retries all of this generates safe, and verify the resulting money records against the outside world. The remaining practices — monitoring, logging, load testing, incident management, retrospectives — are the verification loop wrapped around that chain.

The first bound is time, and de Water calls it the single takeaway if a reader keeps only one: investigate and set low timeouts everywhere you can. Shopify sets them in HTTP clients and in data stores — MySQL's MAX_EXECUTION_TIME optimizer hint puts a per-SELECT ceiling in milliseconds, combined with tools like pt-kill to keep bad queries from overwhelming the database. The right values depend on the application and should be deduced from monitoring, but the post offers a starting point: a one-second open timeout with a five-second write and read (or query) timeout, justified from the buyer's seat — would you wait more than five seconds for a page to either load or show an error?

Timeouts still spend a full timeout per attempt, and services that go down tend to stay down for a while. Circuit breakers close that gap: after multiple timeouts in a short period, the circuit opens and calls fail instantly instead of trying at all. Shopify built Semian to protect Net::HTTP, MySQL, Redis, and gRPC calls in Ruby; raising an exception the moment a service is detected as down saves the resources another expected timeout would burn, and rescuing that exception is where fallbacks live — designed and unit-tested with Toxiproxy. The detail that shows the pattern taken seriously is granularity. Worldwide payment processing typically uses a single gateway endpoint, but gateways often route through local acquirers per country to optimize authorization rates and costs; for Shopify Payments credit card transactions, the merchant's country code is added to the host-and-port Semian identifier, so an open circuit caused by a local outage in one country does not stop transactions for merchants everywhere else. The post is equally clear that breakers are not a silver bullet: they require understanding the ways the application can fail and what falling back should look like, and at scale a circuit breaker can still waste a lot of resources and money — Shopify wrote a separate article on misconfigured breakers.

The third bound is admission. No application can out-scale the world, so at some point a limit must be put on incoming work — rate limiting and load shedding are the named techniques. Shopify's most interesting instrument here is a scriptable load balancer that throttles the number of checkouts happening at any given time: when the buyers wanting to check out exceed capacity, they are placed in a waiting queue before being allowed to pay for their order. Whether the limits actually hold is tested by regularly simulating large-volume flash sales against dedicated benchmark stores. End-to-end payments load testing has a special problem — the test and staging environments of financial partners do not share production's capacity or latency distribution — so the benchmark stores are configured with a benchmark gateway whose responses mimic those properties.

Everything above makes requests fail faster and more often on purpose, which is only tolerable because retries are made safe. Shopify's centralized payment service tracks attempts — consisting of one or more identical, possibly retried API requests — by an idempotency key unique to each attempt. The key looks up which steps the attempt already completed, such as creating the local database record of the transaction, and makes sure only a single request is sent to the financial partners. If any step failed and a retried request arrives under the same key, recovery steps run first to recreate the same state before continuing. A key needs to stay unique only for as long as the request should be retryable, typically 24 hours or less. Even the key format is a scaling decision: Shopify prefers ULIDs — a 48-bit timestamp followed by 80 bits of randomness — over random UUIDv4, because sortable identifiers work far better with the b-tree structures databases use for indexing; in one high-throughput system, switching idempotency keys from UUIDv4 to ULID produced a 50% decrease in INSERT statement duration.

The last line of defense concedes that prevention is incomplete. Reconciliation checks Shopify's records against its financial partners' — individual records like charges and refunds, and aggregates like the balance not yet paid out to a merchant — records that also feed the tax forms Shopify must generate for merchants in some jurisdictions. A mismatch is recorded in the database as an anomaly: a MismatchCaptureStatusAnomaly, for example, expresses that a captured local charge's status disagreed with what the partner returned. Where possible an automatic remediation resolves the discrepancy and marks the anomaly resolved; where it is not, the developer team investigates and ships fixes. The stance is deliberate: track every mismatch so the team knows what the system did and how often, but treat anomalies as a last resort and prefer solutions that prevent them from being created at all.

The verification loop starts with Google's four golden signals — latency, traffic, errors, saturation — with two payments-specific refinements. Failures and errors are distinct: a charge declined for insufficient funds is not unexpected at all, while an HTTP 500 from a financial partner is; and because circuit breakers make failures happen very fast, latency graphs mislead unless broken down between success and failure. Structured logs in machine-readable form carry a correlation identifier across the whole distributed flow — generated in the Rails controller when a buyer initiates payment at checkout, passed to the background job, and into the payment service's API parameters and SQL query comments — so a single payment attempt's story is searchable end to end. When something still goes wrong, an incident begins with a page (automatic or by hand) and a command to the Slack bot spy, and runs with three roles: an Incident Manager on Call coordinating, a Support Response Manager handling public communication, and the service owners restoring stability; afterward the bot generates a Service Disruption record with an initial timeline, Slack messages marked important, and the people involved. A retrospective within the following week digs into what exactly happened, which assumptions about the system were wrong, and what will prevent recurrence; incident writeups are internally public to all employees, and a well-documented one can become a disaster role-playing scenario for training the next engineer joining the on-call rotation.