Thread
#prefect-community
    Kevin Mullins

    Kevin Mullins

    10 months ago
    I’m somewhat new to prefect. I’ve done simple mapped tasks but am curious if I can chain subsequent tasks to mapped tasks to perform a fan-out/fan-in type approach. Here is a diagram trying to visualize what I’d like to accomplish. Any guidance if this is possible and/or approaches would be greatly appreciated:
    Kevin Kho

    Kevin Kho

    10 months ago
    Hi @Kevin Mullins, yes this is definitely doable. You are looking for iterated mapping here that is followed by a reduce step. You don’t need to think too hard about it. Just have a task that takes the list, and it will automatically reduce.
    Kevin Mullins

    Kevin Mullins

    10 months ago
    Interesting. Do you happen to know if the second map iteration has to wait on the first round? For example, I’d like the “Finalize Step 1” task to be able to run as soon as “Load Source 1" completes instead of when all Load Source tasks complete.
    Kevin Kho

    Kevin Kho

    10 months ago
    DaskExecutor will prefer depth-first execution
    Kevin Mullins

    Kevin Mullins

    10 months ago
    Thanks, I’ll give it a go!
    Kevin Kho

    Kevin Kho

    10 months ago
    Local is sequential so breadth first and depth first will just be the same
    Kevin Mullins

    Kevin Mullins

    10 months ago
    does LocalDaskExecutor behave the same as DaskExecutor?
    Kevin Kho

    Kevin Kho

    10 months ago
    Kinda, but a lot of people have reported breadth first execution instead. It has to do with the scale. Dask makes decisions which to run you can’t force it one way or the other. For smaller datasets, you may see more breadth first execution
    Kevin Mullins

    Kevin Mullins

    10 months ago
    Good to know, thanks again!
    Sorry to keep tacking on to this, but another thought I had was to see if it is possible wrap/extend ShellTask. I really just need to be able to load some state to write to a file before running a shell task and then save state from the ShellTask output after it’s finished for the next run. Would it be terrible to inherit from ShellTask and do this before/after the call the super’s run?
    Just trying to brainstorm the best approach for something like this.
    Kevin Kho

    Kevin Kho

    10 months ago
    You can extend all the tasks in the task library for sure and we have a lot of users that do that.
    Kevin Mullins

    Kevin Mullins

    10 months ago
    Awesome, that sounds preferable to me for my use-case. Just wanted to make sure it wouldn’t be bad practice.