Hi,
We have a flow that consists of multiple flows. There are two parent flows on which multiple flows depend. The two parent flows are independent of each other. I see the flows execute serially satisfying dependency constraints... but don't see any nondependent flows running in parallel. Should we expect the two independent flows to be scheduled in parallel or they would still run serially one after the other?
Krapi Shah
08/19/2021, 12:41 PM
@Charles Leung adding you in the loop as well
k
Kevin Kho
08/19/2021, 1:03 PM
Hey @Krapi Shah, are you using LocalDaskExecutor or DaskExecutor? I think it should run in parallel if there is no dependency
c
Charles Leung
08/19/2021, 1:05 PM
Hey Kevin, we didn't change or set any executor - which one would be the default?
k
Kevin Kho
08/19/2021, 1:14 PM
The default is
LocalExecutor
so sequential execution. For any parallelism, you’ll need to set
LocalDaskExecutor
or
DaskExecutor
👍 1
k
Krapi Shah
08/19/2021, 1:30 PM
And in this scenario, any of the two might start first?
k
Kevin Kho
08/19/2021, 1:31 PM
Yes I think any of independent starting tasks can start first
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.