Hi all,
I am experimenting with tasks and subflows in prefect 2.0. While in tasks I can use the wait_for keyword to enforce task dependency, the same keyword does not work for subflows. I did not found much on the subflows docs, but it seems that in prefect 1.0 dependency between subflows was possible. Is such a feature available also in prefect 2 ?
z
Zanie
04/22/2022, 4:51 PM
This just hasn’t been added yet
Zanie
04/22/2022, 4:52 PM
However, since a subflow will block until it is complete, the
wait_for
kwarg is less useful
Zanie
04/22/2022, 4:52 PM
You might as well just wait for all the tasks to finish using
.wait()
before calling the subflow
d
Domenico Di Gangi
04/22/2022, 4:57 PM
Thanks for the quick reply @Zanie.
If I understand correctly, using wait_for in a task also injects a "logical dependency" not only a temporal ordering. For example, in the following
Copy code
@task
def task_a():
pass
@task
def task_b():
pass
@flow
def my_flow():
a = task_a()
b = task_b(wait_for=[a])
if task_a fails, task_b does not execute, right?
I am looking for a similar dependency between subflows or a subflow depending on a task
z
Zanie
04/22/2022, 4:59 PM
Ah yes, I see your point. You can pass the task futures to the subflow directly but when you do not want to add a data dependency this introduces a problem
Zanie
04/22/2022, 5:00 PM
You can add a
wait_for
argument to your subflow that takes a list of upstream tasks and just not use the data if you don’t need it. We’ll likely support this as a first-class feature in the future. I’ll open an issue for tracking.
👍 1
d
Domenico Di Gangi
04/22/2022, 5:01 PM
I think this will work. Many thanks for your help @Zanie
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.