https://prefect.io logo
#prefect-community
Title
# prefect-community
j

Jai P

04/27/2022, 6:29 PM
👋 in a flow that contains subflows in prefect 2.0, is there a way to specify
wait_for
between the subflows? i'll drop a trivial example in the thread
👋 1
let's say i have something like:
Copy code
@task
def my_task():
    ...

@flow
def subflow_1():
    my_task()

@flow
def subflow_2():
    my_task()

@flow
def my_flow():
    subflow_1()
    subflow_2()
i'd like to avoid
my_task
running twice (maybe its a time-intensive task) in parallel
a

Anna Geller

04/27/2022, 6:33 PM
Yes, there is! In fact, if you use the SequentialTaskRunner, subflow_2 won't start until subflow_1 finishes.
Copy code
from prefect.task_runners import SequentialTaskRunner


@flow(task_runner=SequentialTaskRunner())
j

Jai P

04/27/2022, 6:35 PM
ah ok! to add onto that, let's say i now also have another task that can run in parallel...is that possible? so the above example would go to
Copy code
@task
def my_task():
    ...

@task
def my_task_2():
    ...

@flow
def subflow_1():
    my_task()
    my_task_2()

@flow
def subflow_2():
    my_task()
    my_task_2()

@flow
def my_flow():
    subflow_1()
    subflow_2()
and the hope would be that
my_task()
runs once and retrieves from cache once, but then
my_task_2()
can return to being run in parallel
oh, unless i just use the
SequentialTaskRunner
on the task itself?
a

Anna Geller

04/27/2022, 6:42 PM
the task runner can be attached to a flow (it determines how to run tasks in a flow; that's why it can't be set on a task), and you can have a different task runner attached to each subflow to ensure each flow/subflow has the parallelism you need
j

Jai P

04/27/2022, 8:43 PM
sorry for the delay, sounds good! sounds like that just means we'll need to structure our code in such a way to take advantage of some things being in parallel and other things not being in parallel
👍 1
(which i think is totally fine! parallelism is hard 🙂)
a

Anna Geller

04/27/2022, 9:17 PM
Exactly! and if you have any questions about it, let us know, we can certainly help and troubleshoot together
8 Views