I’m somewhat new to prefect. I’ve done simple mapp...
# ask-communityk
Kevin Mullins
10/27/2021, 6:43 PMI’m somewhat new to prefect. I’ve done simple mapped tasks but am curious if I can chain subsequent tasks to mapped tasks to perform a fan-out/fan-in type approach. Here is a diagram trying to visualize what I’d like to accomplish. Any guidance if this is possible and/or approaches would be greatly appreciated:
k
Kevin Kho
10/27/2021, 6:45 PM
Hi @Kevin Mullins, yes this is definitely doable. You are looking for iterated mapping here that is followed by a reduce step. You don’t need to think too hard about it. Just have a task that takes the list, and it will automatically reduce.
k
Kevin Mullins
10/27/2021, 6:47 PMInteresting. Do you happen to know if the second map iteration has to wait on the first round? For example, I’d like the “Finalize Step 1” task to be able to run as soon as “Load Source 1" completes instead of when all Load Source tasks complete.
k
Kevin Kho
10/27/2021, 6:48 PM
DaskExecutor will prefer depth-first execution
👍 1
k
Kevin Mullins
10/27/2021, 6:49 PMThanks, I’ll give it a go!
k
Kevin Kho
10/27/2021, 6:50 PM
Local is sequential so breadth first and depth first will just be the same
k
Kevin Mullins
10/27/2021, 6:51 PMdoes LocalDaskExecutor behave the same as DaskExecutor?
k
Kevin Kho
10/27/2021, 6:52 PM
Kinda, but a lot of people have reported breadth first execution instead. It has to do with the scale. Dask makes decisions which to run you can’t force it one way or the other. For smaller datasets, you may see more breadth first execution
k
Kevin Mullins
10/27/2021, 6:52 PMGood to know, thanks again!
👍 1
Kevin Mullins
10/27/2021, 6:54 PMSorry to keep tacking on to this, but another thought I had was to see if it is possible wrap/extend ShellTask. I really just need to be able to load some state to write to a file before running a shell task and then save state from the ShellTask output after it’s finished for the next run. Would it be terrible to inherit from ShellTask and do this before/after the call the super’s run?
Kevin Mullins
10/27/2021, 6:57 PMJust trying to brainstorm the best approach for something like this.
k
Kevin Kho
10/27/2021, 6:59 PM
You can extend all the tasks in the task library for sure and we have a lot of users that do that.
🎉 1
k
Kevin Mullins
10/27/2021, 7:00 PMAwesome, that sounds preferable to me for my use-case. Just wanted to make sure it wouldn’t be bad practice.
5 Views