Cosimo Alfarano
10/28/2025, 1:13 PMMarvin
10/28/2025, 1:13 PMMarvin
10/28/2025, 1:16 PMfrom prefect import flow, task
from prefect.task_runners import ThreadPoolTaskRunner, ProcessPoolTaskRunner
from prefect_dask.task_runners import DaskTaskRunner
@task
def io_bound(x):
# e.g., API calls / DB queries
return f"io {x}"
@task
def cpu_bound(x):
# e.g., CPU-heavy transforms
return x * x
# Subflow for CPU-heavy tasks
@flow
def cpu_subflow(data):
return [cpu_bound.submit(x) for x in data]
# Parent flow uses threads by default
@flow(task_runner=ThreadPoolTaskRunner())
def parent(data, backend="dask"):
# IO-bound work on threads
io_results = [io_bound.submit(x) for x in data]
# Choose the backend for CPU-heavy tasks
if backend == "dask":
cpu_on_dask = cpu_subflow.with_options(task_runner=DaskTaskRunner())
cpu_results = cpu_on_dask(data)
elif backend == "process":
cpu_on_proc = cpu_subflow.with_options(task_runner=ProcessPoolTaskRunner())
cpu_results = cpu_on_proc(data)
else:
cpu_results = cpu_subflow(data)
return io_results, cpu_results
Notes
- The task runner is set per flow, but you can create “variants” of a subflow at call time with subflow.with_options(task_runner=...).
- Prefer ThreadPoolTaskRunner for IO-bound concurrency, ProcessPoolTaskRunner for pure CPU parallelism, and DaskTaskRunner if you want a distributed cluster.
- If you need different infrastructure (Kubernetes, ECS, etc.) per class of work, deploy those subflows separately and orchestrate them from the parent flow.
References
- Prefect task runners (source): prefect/task_runners.py
- Dask task runner: prefect-dask
- Futures and execution: prefect/futures.py
If you can share which runners/backends you want to mix (e.g., Dask + processes, or threads + Prefect’s distributed task worker), I can tailor an example to your setup.