What Remains: Netflix Conductor and the Case Against Choreography

Netflix Technology Blog Dec 12, 2016Read original

Durable Workflows Embedded vs Centralized Orchestration Choreography vs Orchestration

Netflix's Content Platform Engineering team runs the processes that get a title ready to stream (studio integration, content ingestion, encoding, deployment to the CDN) as async task flows spanning several days across microservices. Historically these were wired together case by case: pub/sub messages, direct REST calls, and databases each holding a fragment of the state. Netflix sums up the problem in one sentence: there was almost no way to systematically answer 'what is remaining for a movie's setup to be complete?' Conductor is the answer as architecture: a central engine where a JSON blueprint defines the flow, a Decider (a state machine) computes what comes next from the blueprint plus the current state, and every workflow can be tracked, paused, resumed, and restarted. Open-sourced after nearly a year in production, it had run 2.6 million process flows, from simple linear jobs to dynamic workflows that ran over multiple days.

The crux

A multi-step process that is partly done has to keep its partial state somewhere. Under choreography, where services coordinate by passing messages to one another with no central conductor, that state lives nowhere in particular, which is to say everywhere. Each service knows only that it consumed one event and emitted another. Netflix names three consequences: the flow is buried across codebases; services are tightly coupled through hidden assumptions about each other; and, sharpest, there is no systematic way to answer what remains for a given title. That last one is exactly the partial-completion question. When a days-long flow stalls partway, some steps are done and some are not, and under choreography nothing can even list which are which, let alone safely pick up where it left off. Conductor makes partial completion something you can query: the blueprint says what the whole process is, the instance records what has finished, and the Decider computes the next step from the two. 'What remains' stops being something you piece together by hand and becomes a simple lookup.

Partial completion under crashes

Interactive

Run a title-setup workflow under choreography, press WHAT REMAINS FOR THIS TITLE?, and watch the system fail to answer its own question. Then kill the encode worker, trace the silent stall by hand, and re-run it under the Decider, where the same kill costs you nothing.

Open the visualization ↓

Problem

These flows are large and long-running by nature. Ingesting content from studio partners, ingesting finished masters (in the film industry's IMF format), setting up a new title, and the encode-and-deploy pipeline are all multi-step, multi-service, and long-running, and several days is normal. The traditional wiring was a mix of pub/sub for events, direct REST calls where a synchronous answer was needed, and databases holding pieces of the state. Each choice made sense on its own, but together they left the whole process with no owner. As the microservices multiplied and the processes grew more complex, visibility into these distributed workflows degraded until a basic question about any in-flight process had no owner.

Left, choreography where no one can say what remains; right, orchestration where a Decider owns the blueprint and answers it. — Under choreography (left), each service reacts to an event and emits the next, so the process lives nowhere and nothing can say what remains. Under orchestration (right), a Decider owns the blueprint and state, so the question is a lookup.

Netflix is honest about what they tried first: an early version built on Amazon's SWF workflow service. The limitations that pushed them off it doubled as the requirements for Conductor, and the full list of what they needed reads like a description of what any workflow engine needs:

the flow defined by a blueprint, rather than as deployed decider code (SWF's model),
tracking and management, with the ability to pause, resume, and restart a flow,
a UI to visualize the flows,
synchronous handling of tasks when needed, not only message-passing,
indexed, searchable inputs and outputs, so instances can be found,
scale to millions of concurrent flows,
a queuing service hidden from the clients,
and flexibility in the transport layer (HTTP, and others).

several days

normal duration for a title-setup workflow: partial completion is the steady state, not the edge case

Solution

The engine's heart is the Decider, a state machine. On each workflow event (a task finishing, a task failing) it combines the workflow's blueprint with its current state, works out the next step, and either schedules the next tasks or marks the workflow done. Underneath, it uses a distributed queue that can hold delayed work (Netflix's own dyno-queues, on its Dynomite store), with Dynomite holding the data and Elasticsearch indexing every run so it can be searched. The storage layer is pluggable, so a traditional database can be swapped in.

The Decider compares blueprint and state on each event and schedules the next tasks or ends the workflow. — On every event, the Decider takes the blueprint (what the process should be) and the instance state (what has finished), compares them, and schedules the next tasks. Partial completion becomes a value the engine reads.

A blueprint is written in JSON. The definition lists the tasks, and each task is either a worker task (done by an application) or a control task run by the engine itself. The built-in control tasks are:

Decision, to branch on a condition,
Fork and Join, to run tasks in parallel and then wait for them,
Sub-Workflow, to call another workflow as a step,
and a plug-in point for teams to add their own custom control tasks.

Because definitions are versioned, a flow already in progress keeps running on the version it started with, while new flows pick up the new version. Each task's behavior is set by its task definition. This is where the handling for failures and stalls lives, the kind of thing that, without an engine, every team had to build for itself. Netflix's sample sets it to retry up to 3 times, with a 1200-second timeout, a fixed 600-second wait between retries, and a policy to fail the whole workflow if a task never answers. All of it is declared per task in configuration, not written into each service. Data flows by wiring: a task's inputs come from the workflow's input or an earlier task's output, so an encode's output becomes the publish task's input without either service knowing the other exists.

A graph: Inspect, Encode, a Fork into Subtitles and Thumbnails, a Join, then Publish, with Encode wired to Publish. — A title-setup workflow drawn out. Inspect and Encode are worker tasks; a Fork runs Subtitles and Thumbnails in parallel and a Join waits for both; then Publish. Encode's output is wired straight into Publish's input.

Workers stay deliberately simple: stateless functions, safe to run more than once, that either expose a REST endpoint or (the preferred shape) run a loop that keeps asking the engine for pending work. That polling flips the pressure the right way. Workers pull only at the rate they can handle, so a backlog builds up in the queue rather than overwhelming them. And the depth of each task's queue becomes a signal the engine can use to add or remove workers. The UI is not a nice extra bolted on at the end; it is the whole point. You can search across workflows by their inputs and outputs, see the blueprint drawn out and the exact path an instance took, and open any task to see its details:

when it was scheduled, picked up, and completed,
why it failed, if it did, and how many times it was retried,
which host ran it,
and its inputs and outputs.

That panel is the direct answer to 'what is remaining for this movie': the question that had no owner now has a screen.

A stalled flow: under choreography the state is scattered across five services; under the orchestrator one screen shows what remains. — The encode worker dies mid-task. Under choreography (left), 'what remains?' is scattered across five services' logs and queues with no owner. Under the orchestrator (right), one screen shows it: inspect done, encode retrying, the rest pending.

Production stats after a year: 2.6 million workflow instances, 100 distinct definitions, 190 unique workers, an average of 6 tasks per definition, and a largest workflow of 48 tasks. The list of things still to come (serverless tasks, tighter integration with container schedulers to autoscale workers, per-task execution logs, authoring blueprints from the UI, support for a standard states language) looks, in hindsight, like a preview of features the rest of the workflow-engine industry would spend the next several years building.

2.6M · 100 · 190

workflow instances, distinct definitions, and unique workers after one year in production: one place to ask what any flow still needs

retryCount: 3

crash recovery as per-task configuration (timeoutSeconds 1200, TIME_OUT_WF) instead of per-service code

Open in full

Tradeoffs

Choreography has real advantages, and Netflix grants them for the simple cases: pub/sub worked fine for the simplest flows. There is no central dependency, no engine to operate, and services stay loosely coupled through the message bus. The downside shows up as the flows get complex: the process itself becomes invisible, written down nowhere, and can be reconstructed only by reading every participant's code and queues.
Orchestration gives you that visibility by putting one engine at the center, and a center is a dependency. The Decider, its queues, and its storage now sit under every business flow, so the engine's uptime and its ability to scale (millions of concurrent flows was an explicit requirement) become preconditions for everyone's work. The choreography world had no single point of failure. Conductor becomes one, and it must not fail.
Defining flows as versioned, inspectable JSON data beat writing them as deployed decider code, which was Netflix's stated reason for leaving SWF. But a JSON blueprint is really a programming language dressed up as data. Decision, Fork, Join, and Sub-Workflow are control flow by another name, and complex logic written in this declarative style can be harder to test and reason about than the ordinary code it replaced.
Blueprint versioning solves the upgrade problem by multiplying the live versions. In-flight instances finish on the version they started with, so days-long flows guarantee that several blueprint versions are running at once, and the workers must stay compatible with all of them.
Polling workers trade a little speed for natural backpressure. Constantly asking for work adds a small pickup delay compared to pushing work at them. But it lets workers consume at their own rate and turns each task's queue depth into an honest measure of load. That is the right trade for day-scale flows, and the wrong one the moment latency and sub-second dispatch matter.
The whole safety net quietly rests on the workers being idempotent, and that part is left to the workers. Retries, timeouts, and restarts are only safe because each worker is meant to be a stateless function that is safe to run more than once. The engine's recovery guarantees depend on a property it cannot check, only assume. The machinery is centralized; the discipline it relies on is still spread across every worker.

Patterns in this article

Durable Workflows
Conductor shows the durable-workflows pattern a decade before it became common: a declarative flow definition, durable per-task state, timeouts and retries as configuration, and pause, resume, and restart as ordinary operations. Its own particular contribution is searchability. By indexing every execution in Elasticsearch, it turns the workflow store into a queryable record of what happened, not just a log for recovery.
Embedded vs Centralized Orchestration
Conductor is the earliest, most basic version of the centralized answer. It argues not against an embedded engine inside each service, but against having no engine at all, only pub/sub between services. The useful way to compare these designs is to ask one question: who owns the state of the process? The individual service, a shared central platform, or (under pure choreography) nobody at all.
Choreography vs Orchestration
The question is whether a multi-service process is coordinated by events each service interprets on its own, or by an engine that owns the process definition and drives the participants. Choreography keeps coupling and central dependencies low, but leaves the process written down nowhere: invisible to query and hard to change. Orchestration makes the process a real, queryable object, at the cost of a central dependency. Netflix draws exactly this contrast in its 'Why not peer to peer choreography?' section.

Also solving this

Other systems in behindscale's Partial completion under crashes class: