Solve It Once: Cadence and the Case for the Central Workflow Engine
Cadence is Uber's answer to a question Airbnb answered the opposite way: where should durable execution live? Airbnb's Skipper embeds the workflow engine as a library inside each service, because a central orchestrator's outage would stop every dependent service at once. Uber built exactly that central orchestrator — and made it a platform serving over a thousand services from T0 to T5, running 12 billion executions and 270 billion actions a month at 99.9% availability, with workflows written as ordinary code in native languages rather than DSLs. The bet: the engine is built once while every workflow is written fresh, so complexity belongs in the platform — and Uber's 2021 internal survey found teams writing 40% less code for the same functionality. The costs are the platform's to carry: a team of 20 running a Tier-0 shared dependency, noisy-neighbor isolation as a permanent discipline, and the database bottleneck under everything.
Crash a worker, take down the orchestrator, ship an engine bug, unleash a noisy tenant — against the embedded and the central architecture side by side, and read who pays for each.
Problem
Strip a modern distributed service down and very little of it is the feature. The post's own inventory of nonfunctional requirements — state persistence, queues, timers, retries, failure recovery, scaling, traffic management, observability — surrounds a small core of unique business logic, and only that core plus its tests is actually specific to the service. Everything else is common. At Uber's scale the commonality is the problem: thousands of services, each facing the same crash-between-steps ambiguity, each assembling its own slightly different answer from databases, queues, and cron. The duplication costs engineering time, and worse, it produces a thousand independently maintained implementations of the hardest guarantees in distributed systems.
The traditional consolidation move — a workflow engine driven by DSLs or configuration files declaring task order and dependencies — trades one problem for another. DSLs keep the orchestrator simple, but the post argues they put simplicity on the wrong side: the orchestration engine is built once, while a unique workflow must be written for every use case. As use cases grow, DSLs either cap what a workflow can express or grow so complicated they stop being practical. Engineers think in programs, not in configuration; and considering the range of things that are workflows — sign-up flows, batch processing, distributed cron, model training, microservice orchestration — the post lands on a blunt premise: any program is a workflow.
So the question isn't whether to consolidate durable execution — it's what shape the consolidation takes, and who carries its risks. A central engine is a shared dependency: every service that leans on it inherits its availability, its capacity limits, and its neighbors. That is precisely the blast radius that led Airbnb to embed Skipper as a library inside each Tier-0 service instead. Uber took the other branch and spent six-plus years making the central bet hold.
Solution
A Cadence workflow is a program in a native language — Go, Java, and others via client SDKs — written against Cadence APIs with minimal limitations compared to pure software. The platform makes that program durable, transactional, and fault-tolerant without the author thinking about it: state survives crashes, activities retry, timers fire across days or months, and the workflow resumes from where it left off. The developer-facing surface covers the lifecycle: start, signal, schedule (distributed cron), terminate, cancel; hierarchical parent-child workflows for composing services; searchable attributes to filter among billions of instances; and a UI exposing inputs, outputs, call timelines, and workflow relationships for debugging.
The platform side is where the central bet is paid for. Cadence ships the machinery a shared Tier-0 dependency demands: built-in rate limits, hot-shard detection, and per-domain scale tracking to prevent, detect, and manage noisy neighbors; load balancing and scaling per cluster, domain, workflow, activity, and tasklist; configurable failure behavior including automatic retries, region failovers, and workflow resets that rewind execution to a healthy point after a bad deployment. Versioning manages new code against workflows already in flight — a problem embedded engines simply don't have at platform scale — and a testing story built on replaying and shadowing production workflows catches inconsistencies before they ship.
The operating model is deliberate: major features are built and run internally for months, then ported to open source, keeping every release backward compatible. A core team of 20 engineers carries the platform, with internal teams contributing to Cadence and its substrate — Cassandra, Elasticsearch, Kafka, MySQL — and external companies (DoorDash, HashiCorp, Coinbase run it) maintaining their own Cadence teams.
The results are the argument. Over a thousand services at Uber, from T0 most-critical downward, run on Cadence: long-running workflows, synchronous interactions, microservice orchestration, batch processing, distributed cron, distributed singletons, data pipelines, model training. The platform executes 12 billion workflow executions and 270 billion actions a month at Uber alone, has sustained 100% year-over-year growth for several years, and guarantees 99.9% availability. Uber's 2021 internal survey found teams writing 40% less code to implement the same functionality — the number that makes the whole consolidation defensible.
Tradeoffs
- The central engine is the blast radius Airbnb refused. A thousand services from T0 down share one platform's fate: its availability guarantee — 99.9% — is now a ceiling inherited by everything above it, and a bad day for Cadence is a bad day for rides, payments, and every pipeline in between. Uber's answer isn't to deny the coupling but to industrialize it: a dedicated 2022 reliability year, region failovers, capacity management, and a platform team whose entire job is keeping the shared dependency boring. The divergence with Skipper is not about who understood the risk — both did — it's about whether an organization would rather operate one hard thing extremely well or distribute the hardness into every service.
- Multi-tenancy makes noisy neighbors a permanent discipline, not an incident type. When a thousand tenants share queues, shards, and a database, one team's burst — a million timers firing at once — is everyone's latency. The feature list betrays how much of the platform exists for this: built-in rate limits, hot-shard detection, per-domain tracking, capacity management by rate limits. Isolation that an embedded engine gets by construction (your engine only runs your workflows), the central engine must build, tune, and operate forever.
- Determinism is the tax on writing workflows as ordinary code. The replay model that makes workflows durable requires re-execution to reach the same decisions, which makes nondeterministic code — and even routine code changes to in-flight workflows — a correctness hazard rather than a style concern. The platform grows machinery in proportion: versioning for behavior changes against running workflows, replay and shadowing of production traffic to detect inconsistencies, and a roadmap item to catch nondeterministic changes at development time, which is an admission of where they're caught today. The 'any program is a workflow' pitch is real, but the program lives under constraints ordinary programs don't.
- Backward compatibility compounds into drag. Every Cadence release has stayed backward compatible for six-plus years, which is exactly what a thousand dependent services require — and it means API mistakes cannot be fixed, only worked around, which the post concedes is a challenge when issues surface in new APIs. The planned V2 branch exists precisely because some changes cannot happen compatibly. A central platform doesn't just inherit its tenants' load; it inherits their inertia.
- The database under the platform is the bottleneck under everything. The post says it plainly: DB load 'seems to be the bottleneck most of the time,' capacity is managed by rate limits, and the efficiency roadmap — evenly distributed load by design, new workflow modes, new capacity management — is a plan to buy back headroom. Centralizing a thousand services' durable state concentrates their write load into one storage estate; the consolidation that removes duplicated machinery also concentrates the scaling problem it must now solve on everyone's behalf.
- A 20-engineer platform team is the visible price of 'solve it once.' The 40%-less-code figure is measured across consuming teams; the denominator includes a permanent, growing core team, internal contributors across Cassandra, Elasticsearch, Kafka, and MySQL, and — at partner companies — dedicated Cadence teams of their own. The economics favor the platform at Uber's scale and use-case density, which is the honest boundary of the lesson: the central bet is not that platforms are free, it's that at sufficient scale the duplicated alternative costs more.
Patterns in this article
- Durable Workflows
Cadence delivers the same guarantee Skipper delivers — multi-step programs that survive crashes and resume with committed work intact — through the same fundamental mechanism family (persisted state, replayable execution, versioning for in-flight changes). The recurrence across opposite topologies is the strongest evidence the pattern is real: the guarantee is the invariant; where the engine lives is the variable.
- Embedded vs Centralized Orchestration
This article is the pattern's other pole. Skipper instantiates embedded: engine-as-library, no shared dependency, blast radius contained to one service — at the cost of shipping and operating the engine everywhere. Cadence instantiates centralized: one platform, one team, one availability contract shared by a thousand tenants — at the cost of noisy-neighbor discipline, backward-compatibility drag, and a concentrated database bottleneck. Same crux, opposite answers, both correct for their constraints.
Also solving this
Other systems in behindscale's Partial completion under crashes class: