Is there a trick in prefect to get a map of a map to parallelise correctly
We've created a flow and it appears as though a map within a map isn't working as expected.
I can produce an example but just wanted to know if this is a common problem.
10/19/2020, 3:53 PM
Hey @Ralph Willgoss,
I don't believe there are recommended practices for a map of a map, particularly for resource allocation on a Dask cluster - is there a specific issue you keep running into with your implementation? I am also definitely interested if others have an example they are working with.
10/19/2020, 3:54 PM
thanks @Kyle Moon-Wright yeah there is a specific issue
we've constructed a flow and its not paralleling as we would expect.
10/19/2020, 4:00 PM
I'm guessing this would be the map of the map is not parallelizing as you're expecting? Are you able to parallelize the higher level map at least? I'll do some testing on my end to better understand this use case.
10/19/2020, 4:13 PM
yes we are
i have an example that I'll post tomorrow and tag for you if thats ok
@Kyle Moon-Wright i actually have the visualization for it
not sure if that helps without the code. we would expect individual_preprocessing to parallelise as well but its not. thanks!
10/19/2020, 4:25 PM
Thanks! Definitely interested in the code for this when it's ready/sanitized.