Netflix has upgraded its Conductor workflow engine to support workflows with up to 30,000 tasks, a significant increase from the previous 2,500 limit. The update reduces p99 evaluation latency by approximately 40% by decoupling workflow metadata from task data and shifting evaluation to asynchronous processing. Key architectural changes include dynamic worker allocation and new concurrency controls to handle 420 million monthly executions.
- Workflow size limit increased from 2,500 to 30,000 tasks, enabling complex multi-step pipelines.
- P99 latency drops 40% via async processing and metadata/task data separation.
- Dynamic worker allocation and concurrency controls manage 420M monthly executions.
- Architectural shift to async evaluation supports larger, more intricate workflow definitions.