<@ULVA73B9P> is it possible to use different task ...
# ask-marvin
c
@Marvin is it possible to use different task runners in the same flow? I have tasks which should be run on selected runners, and other in other runners. maybe have a default runner, but still have control over where to dispatch a task, on a per task or per class of task basis
m
thought for 194.8 seconds
Short answer: not within a single flow run. In Prefect 3.x, the task runner is a flow-level setting, so all tasks inside one flow use the same runner. There isn’t a per-task task runner override. Good patterns to get what you want: - Subflows with different task runners - Group tasks by “class” (CPU-heavy, IO-heavy, distributed, etc.) into subflows. - Give each subflow its own task runner and call the appropriate subflow from your parent flow. - Deployments per backend (optional) - If you need different infrastructure/work pools, deploy those subflows separately and orchestrate them from a parent flow. Example: one parent flow, with IO on threads and heavy compute on Dask or processes
Copy code
from prefect import flow, task
from prefect.task_runners import ThreadPoolTaskRunner, ProcessPoolTaskRunner
from prefect_dask.task_runners import DaskTaskRunner

@task
def io_bound(x):
    # e.g., API calls / DB queries
    return f"io {x}"

@task
def cpu_bound(x):
    # e.g., CPU-heavy transforms
    return x * x

# Subflow for CPU-heavy tasks
@flow
def cpu_subflow(data):
    return [cpu_bound.submit(x) for x in data]

# Parent flow uses threads by default
@flow(task_runner=ThreadPoolTaskRunner())
def parent(data, backend="dask"):
    # IO-bound work on threads
    io_results = [io_bound.submit(x) for x in data]

    # Choose the backend for CPU-heavy tasks
    if backend == "dask":
        cpu_on_dask = cpu_subflow.with_options(task_runner=DaskTaskRunner())
        cpu_results = cpu_on_dask(data)
    elif backend == "process":
        cpu_on_proc = cpu_subflow.with_options(task_runner=ProcessPoolTaskRunner())
        cpu_results = cpu_on_proc(data)
    else:
        cpu_results = cpu_subflow(data)

    return io_results, cpu_results
Notes - The task runner is set per flow, but you can create “variants” of a subflow at call time with
subflow.with_options(task_runner=...)
. - Prefer ThreadPoolTaskRunner for IO-bound concurrency, ProcessPoolTaskRunner for pure CPU parallelism, and DaskTaskRunner if you want a distributed cluster. - If you need different infrastructure (Kubernetes, ECS, etc.) per class of work, deploy those subflows separately and orchestrate them from the parent flow. References - Prefect task runners (source): prefect/task_runners.py - Dask task runner: prefect-dask - Futures and execution: prefect/futures.py If you can share which runners/backends you want to mix (e.g., Dask + processes, or threads + Prefect’s distributed task worker), I can tailor an example to your setup.