<@ULVA73B9P> I need to ensure the tasks of my flow...
# ask-marvin
a
@Marvin I need to ensure the tasks of my flow run on the same device sequentially. How can I do this ?
m
thought for 118.9 seconds
Great question. Assuming Prefect 3.x, there are two parts to this: 1) keep all task work on the same machine, and 2) make execution strictly sequential. By default, all tasks in a flow run execute on the same machine that is running the flow run (i.e., the worker that picked up the flow). Tasks only hop to other machines if you’re using Task Workers to offload task execution. To force sequential execution, use one of the approaches below. Option A — Call tasks directly (simplest, fully sequential on the same device)
Copy code
from prefect import flow, task

@task
def step1():
    ...

@task
def step2(x):
    ...

@flow
def my_flow():
    a = step1()         # runs and completes
    b = step2(a)        # runs after step1 on the same machine
    return b
Option B — If you use .submit(), use a single-worker task runner - Thread-based (good for I/O-bound tasks):
Copy code
from prefect import flow, task
from prefect.task_runners import ThreadPoolTaskRunner

@task
def step(i):
    ...

@flow(task_runner=ThreadPoolTaskRunner(max_workers=1))
def my_flow():
    a = step.submit(1)
    b = step.submit(2)   # queued behind a; executes sequentially
    return b.result()
- Process-based (good for CPU-bound tasks):
Copy code
from prefect import flow, task
from prefect.task_runners import ProcessPoolTaskRunner

@task
def cpu_step(i):
    ...

@flow(task_runner=ProcessPoolTaskRunner(max_workers=1))
def my_flow():
    a = cpu_step.submit(1)
    b = cpu_step.submit(2)   # queued behind a; executes sequentially
    return b.result()
Notes - Same device: As long as you’re not using Task Workers to offload tasks, everything runs in the flow run’s process on the machine that picked up the flow. - If you are using Task Workers and still need “same device,” either: - Don’t offload tasks for this flow (run them in-process), or - Run a single Task Worker instance on the target device and route tasks to it (happy to advise if you share your setup). Quick clarifier to make sure we tailor this correctly: - Are you using
.submit()
anywhere in your flow? - Are you using Task Workers (offloading tasks) or just running tasks in-process with the flow run?
n
@Marvin using max_workers is a bit more indirect than just using wait_for no?
m
thought for 25.4 seconds
Short answer: it depends on your goal. - If you just need to enforce order between a few tasks, using wait_for (or passing upstream results) is more direct and expressive. - If you want “never run more than one task at a time” across the entire flow, setting max_workers=1 is simpler and more robust than wiring wait_for everywhere. Key differences - wait_for: Adds explicit dependencies but does not reduce overall capacity. Any task you don’t chain can still run in parallel.
Copy code
a = step1.submit()
b = step2.submit(wait_for=[a])   # b won’t start until a finishes
c = step3.submit()               # will run concurrently with a or b unless also chained
- max_workers=1: A blanket guarantee that only one task runs at a time in this flow run, regardless of dependencies. Great for dynamic loops/mapped tasks where adding wait_for everywhere is cumbersome.
Copy code
from prefect.task_runners import ThreadPoolTaskRunner

@flow(task_runner=ThreadPoolTaskRunner(max_workers=1))
def my_flow():
    a = step1.submit()
    b = step2.submit()
    c = step3.submit()
    # These will execute strictly one-by-one on the same machine
Practical guidance - Want strict sequential behavior across the whole flow? Use max_workers=1 (or avoid .submit() and just call tasks directly). - Want some parallelism but specific ordering between certain tasks? Keep higher max_workers and use wait_for (or pass upstream results) where needed. Either way, tasks stay on the same device by default if you’re not offloading to Task Workers. If you are offloading tasks and also need “same device,” we can cover routing + concurrency limits as well.