Hello,
If I have a flow with DaskTaskRunner, but some tasks have parameter dependencies between them, and some other tasks have specified dependency using "wait_for", so essentially the tasks in the flow can be divided into a group of 3, and each group can run in parallel.
DaskTaskRunner recognizes that right? I just want to make sure if having any dependency in there affects parallelism even for tasks that could run in parallel?
I didn't want to separate them into 3 subflows because the logic is better as one flow, but wondering if I have to.
✅ 1
k
Khuyen Tran
10/13/2022, 2:03 PM
Can you provide a minimal example of your use case?
So what I am anticipating is, the 2 functions run sequentially, then the first 7 tasks run in parallel, then the 3 tasks after them run in parallel, then the last 4 tasks run in parallel.
k
Khuyen Tran
10/13/2022, 3:26 PM
It looks like it will behave as you describe. But you can double check with the logs to make sure it works as expected
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.