Hi all, we are waiting with much anticipation for https://github.com/PrefectHQ/prefect/issues/2041 ! It will improve performance for us a lot. Having said that I'm wondering if there is a pattern / feature that will come along with this to allow yielding results to the DFE? We have a task that is recursive in nature (walking a tree) that we implement using the LOOP pattern - wonder if there is something coming that will allow one loop to spawn multiple loops concurrently (I guess this is a separate issue), and then start processing the result of each loop in DFE way?
05/01/2020, 2:22 PM
Hi @Jacques - thanks for the input! I'll need to check with the core team about what #2041 will include - let me get back to you.
05/01/2020, 2:33 PM
05/01/2020, 2:38 PM
Hello again @Jacques - First of all a caveat that we're not certain DFE will allow for this --> tasks must finish before downstream tasks begin, so “yielding” values off a task while it’s still running is an issue.
I'm not sure exactly 😅. I haven't used the DFE yet (is there somewhere I can try it?) so not sure of the use-case here exactly.
Ok, so without understanding how DFE will work exactly, perhaps the solution might be something like: don't loop a task, instead have some mechanism for a task to call .map on itself. Then one task finishes, and can start more of the same task recursively?
This recursive "fan-out" pattern is very common when you do any type of spidering or data crawling - so I wonder if it doesn't make sense to add a feature request?
05/01/2020, 3:00 PM
Ha ha! Fair point since DFE is still in development. If you wouldn't mind adding a feature request setting out what you've said here, that would be great.