Hi all, we are waiting with much anticipation for ...
# ask-communityj
Jacques
05/01/2020, 1:46 PMHi all, we are waiting with much anticipation for https://github.com/PrefectHQ/prefect/issues/2041 ! It will improve performance for us a lot. Having said that I'm wondering if there is a pattern / feature that will come along with this to allow yielding results to the DFE? We have a task that is recursive in nature (walking a tree) that we implement using the LOOP pattern - wonder if there is something coming that will allow one loop to spawn multiple loops concurrently (I guess this is a separate issue), and then start processing the result of each loop in DFE way?
j
Jenny
05/01/2020, 2:22 PMHi @Jacques - thanks for the input! I'll need to check with the core team about what #2041 will include - let me get back to you.
j
Jacques
05/01/2020, 2:33 PMThanks!
j
Jenny
05/01/2020, 2:38 PMHello again @Jacques - First of all a caveat that we're not certain DFE will allow for this --> tasks must finish before downstream tasks begin, so “yielding” values off a task while it’s still running is an issue.
Jenny
05/01/2020, 2:40 PMBut also wanted to check if issue #2091 - Implement a
flat_mapj
Jacques
05/01/2020, 2:47 PMI'm not sure exactly 😅. I haven't used the DFE yet (is there somewhere I can try it?) so not sure of the use-case here exactly.
Jacques
05/01/2020, 2:50 PMOk, so without understanding how DFE will work exactly, perhaps the solution might be something like: don't loop a task, instead have some mechanism for a task to call .map on itself. Then one task finishes, and can start more of the same task recursively?
Jacques
05/01/2020, 2:54 PMThis recursive "fan-out" pattern is very common when you do any type of spidering or data crawling - so I wonder if it doesn't make sense to add a feature request?
j
Jenny
05/01/2020, 3:00 PMHa ha! Fair point since DFE is still in development. If you wouldn't mind adding a feature request setting out what you've said here, that would be great.
j
Jacques
05/01/2020, 3:59 PMCool, done so here: https://github.com/PrefectHQ/prefect/issues/2465
j
Jenny
05/01/2020, 4:03 PMGreat. Thank you!