Hi there, another person evaluating Prefect here (it's going very well, btw). I saw https://github.com/PrefectHQ/prefect/pull/2646, which is great since it directly addresses one of my use-cases as well. I'm curious to know if that would also work in conjunction with the

LOOP

construct. The scenario I have in mind is that I have one task that is generating a series of chunks of data. Each chunk would ideally start to be processed by a different downstream task before the generator task completes. I don't know how many it will create in advance, so I don't think I can

map

the task. (It's also possible that I'm just thinking about the problem the wrong way.)

Hi @Skip Breidbach and welcome! Mapping and Looping can certainly be used in conjunction. There is one thing you said that I want to call out though - in Prefect, each task has to finish before its downstream dependencies can begin running. So as long as your design takes this into consideration you should be good to go!

Thanks, @Chris White! That makes sense to me regarding task completion. My understanding is that depth-first-execution works with mapping because each mapped task is scheduled independently. From the documentation (and experimentation) it looks like the

LOOP

construct also creates child tasks. Would those child tasks also work with depth-first-execution? (Or does the entire

LOOP

need to complete before any dependent tasks could begin?)

Yup your understanding is correct on mapped tasks being scheduled independently; for looping w/ mapping, your entire loop will need to finish before the next task can begin

Bummer, but understandable. Are there any patterns for a task that generates a stream of output that needs downstream processing?

@Skip Breidbach I think I face a similar situation. Did you make any progress on this. I have a call to a service to return data, but I don't know how many calls I have to make, so it's a while loop and I am using the

LOOP

construct. But I would like to be able to return the value of particular iteration to downstream tasks for processing and then combine the result.

@Hawkar Mahmod I didn't - as far as I know the

LOOP

result is an atomic unit. That example appears to show executing one

Task

that internally uses

LOOP

multiple times (potentially in parallel) via

map

. But I don't think it's letting you see the result of a specific

LOOP

iteration, only the final result.