https://prefect.io logo
r

Ralph Willgoss

10/19/2020, 2:28 PM
Hi, Is there a trick in prefect to get a map of a map to parallelise correctly We've created a flow and it appears as though a map within a map isn't working as expected. I can produce an example but just wanted to know if this is a common problem.
k

Kyle Moon-Wright

10/19/2020, 3:53 PM
Hey @Ralph Willgoss, I don't believe there are recommended practices for a map of a map, particularly for resource allocation on a Dask cluster - is there a specific issue you keep running into with your implementation? I am also definitely interested if others have an example they are working with.
r

Ralph Willgoss

10/19/2020, 3:54 PM
thanks @Kyle Moon-Wright yeah there is a specific issue we've constructed a flow and its not paralleling as we would expect.
k

Kyle Moon-Wright

10/19/2020, 4:00 PM
I'm guessing this would be the map of the map is not parallelizing as you're expecting? Are you able to parallelize the higher level map at least? I'll do some testing on my end to better understand this use case.
r

Ralph Willgoss

10/19/2020, 4:13 PM
yes we are i have an example that I'll post tomorrow and tag for you if thats ok
👍 1
@Kyle Moon-Wright i actually have the visualization for it not sure if that helps without the code. we would expect individual_preprocessing to parallelise as well but its not. thanks!
k

Kyle Moon-Wright

10/19/2020, 4:25 PM
Thanks! Definitely interested in the code for this when it's ready/sanitized.
👍 1
r

Ralph Willgoss

10/20/2020, 1:43 PM
@Kyle Moon-Wright Hi, Please find the demo here: https://github.com/ralphwillgoss/prefect/tree/main/parallel Any questions let me know