Hi all we are waiting with much anticipation for

Question

Hi all we are waiting with much anticipation for <https github com PrefectHQ prefect issues 2041> It will improve performance for us a lot Having said that I m wondering if there is a pattern feature that will come along with this to allow yielding results to the DFE We have a task that is recursive in nature walking a tree that we implement using the LOOP pattern wonder if there is something coming that will allow one loop to spawn multiple loops concurrently I guess this is a separate issue and then start processing the result of each loop in DFE way

Jenny · Answer

Hi < Jacques> thanks for the input I ll need to check with the core team about what 2041 will include let me get back to you

Jacques · Answer

Thanks

Jenny · Answer

Hello again < Jacques> First of all a caveat that we re not certain DFE will allow for this gt tasks must finish before downstream tasks begin so yielding values off a task while it s still running is an issue

Jenny · Answer

But also wanted to check if <https github com PrefectHQ prefect issues 2091|issue 2091> Implement a `flat map` operator that s compatible with DFE might help

Jacques · Answer

I m not sure exactly sweat smile I haven t used the DFE yet is there somewhere I can try it so not sure of the use case here exactly

Jacques · Answer

Ok so without understanding how DFE will work exactly perhaps the solution might be something like don t loop a task instead have some mechanism for a task to call map on itself Then one task finishes and can start more of the same task recursively

Jacques · Answer

This recursive fan out pattern is very common when you do any type of spidering or data crawling so I wonder if it doesn t make sense to add a feature request

Jenny · Answer

Ha ha Fair point since DFE is still in development If you wouldn t mind adding a feature request setting out what you ve said here that would be great

Jacques · Answer

Cool done so here <https github com PrefectHQ prefect issues 2465>

Jenny · Answer

Great Thank you