simone
09/17/2020, 2:02 PMout = func.map([A1, A2, A3, A4, A5, B1, B2, B3, B4, B5])
in the next step I would like to partially reduce the output and combine only the matching subgroups ex combining A = out[0:5] B =out[5::]
and then process in parallel A
and`B`.
I have three questions:
(1) If I understood correctly order matters for mapping in prefect so input and output have the same order, correct?
(2) I am running the code on a HPC. If I proceed this way will the entire out
be collected in memory or the different output groups dispatched to the specific worker where the reduce is happening?
(3) Is there a more efficient way to do this?
thanks a lot!Dylan
out[5::]
because the result of the map doesn’t instantiate until runtimefilter
from the control flow utilities insteadsimone
09/17/2020, 5:39 PMDylan
simone
09/17/2020, 6:26 PMDylan
simone
09/17/2020, 7:09 PM