HI #prefect-community I have task mapping on a list of images (~12000) :

out = func.map([A1, A2, A3, A4, A5, B1, B2, B3, B4, B5])

in the next step I would like to partially reduce the output and combine only the matching subgroups ex combining 

A = out[0:5] B =out[5::]

and then process in parallel 

A

 and`B`. I have three questions: (1) If I understood correctly order matters for mapping in prefect so input and output have the same order, correct? (2) I am running the code on a HPC. If I proceed this way will the entire 

out

 be collected in memory or the different output groups dispatched to the specific worker where the reduce is happening? (3) Is there a more efficient way to do this? thanks a lot!

Hi Simone, Here are some control flow resources: https://docs.prefect.io/core/task_library/control_flow.html

Thanks a lot! I will look into filtering. i guess work case scenario i will have ~12000 files

You can always clean then up as part of your flow!

so just out of curiosity you do not thing that this much io will affect the speed of the processing? The flow runs on a SSD drive

It definitely will

great! thanks a lot for the thorough explanation! really appreciated! I will play around and see what is the best solution for my application.