Skipper: Building Airbnb's Embedded Workflow Engine

Airbnb runs many multi-step business processes (insurance claims, payments, media processing, infrastructure automation) that span minutes to days. Take a claim: it might validate, run trust-and-safety checks, assess estimates, process a payout, and notify the host. If the server crashes after validation but before the payout, a traditional design leaves the outcome to chance: a timeout that makes the caller retry, running the work twice, or partial state that corrupts what follows. When a process waits for hours or days, being interrupted partway is not a rare edge case; it is the normal thing to plan for.

The industry already has answers, and Airbnb weighed three:

• A dedicated orchestration cluster (systems like Temporal or Cadence): a separate shared cluster that runs the workflows. It is reliable and gives exactly-once guarantees, but it is a new critical dependency, and for Airbnb's most critical services (the ones directly behind user-facing transactions) that was disqualifying: one cluster outage would stop every dependent service from starting or advancing a workflow.
• A cloud-managed workflow service: the same shared-dependency concern, plus vendor lock-in and data-handling constraints.
• A homegrown queue-based system: it avoids the outside dependency, but every team ends up building its own tangle of retries, state, and cleanup.

A LIBRARY, NOT A SHARED CLUSTER

There is also a second, less obvious problem, and it is common across the industry. When a multi-step process is wired together from queues, scheduled jobs, callback endpoints, and reconciliation scripts, the business logic gets fragmented. There is no single place in the code that says what the process actually does, and every fragment tangles the business rules together with retry timing, duplicate-suppression, and timeout handling. Teams were each rebuilding this, rediscovering the same crash-recovery edge cases, and shipping their own subtle bugs.

Skipper ships as a library embedded directly inside each service. There is no central cluster. Workflow state lives in the database the service already uses (MySQL or Airbnb's internal Unified Data Store), and the engine runs inside the service on its own threads.

The programming model is built from a few simple pieces:

• A Workflow is a plain Kotlin or Java class whose method spells out the process itself: the order of steps, the conditionals, the waits. It reads like the business process it represents.
• An Action wraps a single side effect (an API call, a database write, a notification). A single `@Execute(checkpoint = true)` annotation saves that action's result to the database after it runs (a checkpoint).
• `@StateField` fields hold the workflow's state.
• `@SignalMethod` lets an outside event (a reviewer's decision, a callback) push data into a running workflow, updating the state fields that `waitUntil` conditions check against.

Durability comes from replay. Skipper runs the workflow method and checkpoints each action's result. When execution resumes (after a signal, a timer, or a crash), Skipper replays the workflow method from the top. Previously completed actions do not run again; they return their saved results instantly, and the method picks up where it left off. One important choice: Skipper saves the state fields directly, rather than rebuilding them by replaying a full history of past events the way some other engines do. There is just the current state and the saved action results. That keeps execution lean, especially for workflows with many signals or long histories, in exchange for giving up some ability to trace back, step by step, exactly what happened.

DURABILITY BY REPLAY

A workflow that has to wait goes to sleep. When a workflow hits `waitUntil`, its state is written to the database and the thread is freed. The workflow then exists only as a row in the database until a signal arrives or the timeout fires. It uses zero compute and does no polling, whether the wait is seconds or weeks.

When a workflow fails in mid-run, earlier actions have already taken effect. For example, a listing's photos passed their content check, but then the listing never got published, so that check was done for nothing. Skipper makes undoing that work a built-in feature. The `@Compensate` annotation pairs each action with an undo method, and on failure Skipper runs the undos in reverse order (release inventory, refund charges, revert state), walking the system back to a consistent state. You get eventual consistency without database-spanning transactions, written as 'what does undo mean for this action' rather than manual cleanup jobs.

The best part of the design is the happy path. Most engines add overhead on every run: a central orchestrator needs a network round-trip per step to save its result before moving on. Skipper flips this around. At the start it does two things in the database: it creates the workflow instance, and it schedules a delayed timeout task as a safety net. Then the workflow runs entirely inside the service. Actions execute as ordinary method calls on an in-memory queue, checkpoints are batched, and the workflow runs to completion with no further coordination. The delayed task is what makes this safe: if the process crashes mid-run, a scheduler picks the workflow up after a short waiting period (long enough to be sure the original run really died) and replays it; if the workflow finishes normally, the timeout fires harmlessly and is thrown away.

So on the happy path Skipper adds just a few database writes. The engine only steps in when something goes wrong:

• a crash triggers a replay,
• a `waitUntil` hibernates the workflow,
• an error runs the compensations.

Durability is guaranteed, but you only pay for it when you need it, which is what makes an embedded engine practical inside fast, high-traffic services. Production bears this out. Skipper has run for more than a year, powering 15+ use cases across insurance, payments, media, infrastructure, incentives, and wallet teams. At peak it has scaled to 10,000 workflows per second on DynamoDB, which its lean, coordination-free execution is what makes possible.

PAY ONLY WHEN SOMETHING GOES WRONG