Skipper: Building Airbnb's Embedded Workflow Engine

Airbnb runs many multi-step business processes — insurance claims, payments, media processing, infrastructure automation — that span minutes to days. A claim might validate, run trust-and-safety checks, assess estimates, process a payout, and notify the host; if the server crashes after validation but before the payout, a traditional architecture leaves the outcome to chance: a timeout the caller retries into duplicate processing, or partial state that corrupts what follows. For waits measured in hours or days, such interruptions aren't edge cases — they're expected.

The industry's answer is dedicated orchestration clusters (Temporal, Cadence) or cloud-managed workflow services. Both deliver exactly-once semantics and battle-tested reliability, but both add a critical external dependency. For Airbnb's Tier-0 services — the ones directly behind user-facing transactions — that was the disqualifying cost: an orchestration cluster outage would mean every dependent service loses the ability to start or advance workflows. Cloud-managed services carry the same shared-dependency concern plus vendor lock-in and data-handling constraints. Homegrown queue-based systems avoid the external dependency but trade it for bespoke complexity in every team.

Beneath the infrastructure problem sat a subtler, industry-wide one. When a multi-step process is wired together from queues, scheduled jobs, callback endpoints, and reconciliation scripts, the domain logic fragments — there is no single place in the code that says what the business process does, and every fragment tangles business rules together with retry backoff, deduplication, and timeout handling. Teams were each rebuilding this, rediscovering the same idempotency and crash-recovery edge cases, and shipping their own subtle bugs.

Skipper ships as a library embedded directly inside each service. There is no central cluster: workflow state lives in the database the service already uses — MySQL or Airbnb's internal Unified Data Store (UDS) — and the engine runs in-process on its own thread pools.

The programming model is two abstractions. A Workflow is a plain Kotlin/Java class whose method expresses the orchestration — the order of steps, the conditionals, the waits — and reads like the business process it represents. Actions wrap side effects (API calls, database writes, notifications); a single `@Execute(checkpoint = true)` annotation makes an action's result checkpointed to the database after it runs. Two more primitives round out the model: `@StateField`-annotated fields hold workflow state, and `@SignalMethod` lets external events (a reviewer's decision, a callback) push data into a running workflow, updating the state fields that `waitUntil` conditions evaluate against.

Durability comes from replay. Skipper executes the workflow method and checkpoints each action's result. When execution resumes — after a signal, a timer, or a crash — Skipper replays the workflow method from the top, but previously completed actions don't re-execute; they return their checkpointed results instantly, and the method picks up where it left off. Crucially, Skipper persists state fields directly rather than reconstructing them from an event log: unlike event-sourced orchestrators that replay an entire event history, there is just current state and checkpointed action results. That makes execution leaner — especially for workflows with many signals or long histories — at the cost of some auditability.

Long waits hibernate. When a workflow hits `waitUntil`, its state serializes to the database, the thread returns to the pool, and the workflow exists only as a row until a signal arrives or the timeout fires — zero compute, no polling, whether the wait is seconds or weeks.

When a workflow fails partway through, earlier actions have already taken effect — a dangling content-validation result for a publication that now won't happen. Skipper makes compensation a first-class primitive: the `@Compensate` annotation pairs each action with an undo method, and on failure Skipper runs the compensations in reverse order (release inventory, refund charges, revert state), walking the system back to consistency. It's eventual consistency without distributed transactions, expressed as 'what does undo mean for this action' rather than hand-rolled cleanup jobs.

The design's signature move is the happy path. Most engines impose overhead on every execution — a central orchestrator needs a network round-trip per activity to persist its result before advancing. Skipper inverts this. At start, it does two database things: creates the workflow instance and schedules a delayed timeout task as a durability guarantee. Then the workflow runs entirely in-process — actions execute as ordinary method calls on an in-memory queue, checkpoints are batched — and runs to completion with no further coordination. The delayed task is the safety net: if the process crashes mid-execution, a persistent scheduler picks the workflow up after a lease expires and replays it; if the workflow completes normally, the timeout fires harmlessly and is discarded. The result is that on the happy path Skipper adds just a few database writes — the engine is invoked only when something goes wrong: a crash triggers replay, a `waitUntil` hibernates, an error invokes compensation. Durability is guaranteed, but you only pay for it when you need it — which is what makes an embedded engine viable inside latency-sensitive, high-throughput services. That viability is borne out in production: Skipper has run for more than a year powering 15+ use cases across insurance, payments, media, infrastructure, incentives, and wallet teams, and at peak has scaled to 10,000 workflows per second on DynamoDB — throughput the lean, no-coordination execution model is what enables.