Hi, Is there a trick in prefect to get a map of a map to parallelise correctly We've created a flow and it appears as though a map within a map isn't working as expected. I can produce an example but just wanted to know if this is a common problem.

Hey @Ralph Willgoss, I don't believe there are recommended practices for a map of a map, particularly for resource allocation on a Dask cluster - is there a specific issue you keep running into with your implementation? I am also definitely interested if others have an example they are working with.

thanks @Kyle Moon-Wright yeah there is a specific issue we've constructed a flow and its not paralleling as we would expect.

I'm guessing this would be the map of the map is not parallelizing as you're expecting? Are you able to parallelize the higher level map at least? I'll do some testing on my end to better understand this use case.

yes we are i have an example that I'll post tomorrow and tag for you if thats ok

Thanks! Definitely interested in the code for this when it's ready/sanitized.

@Kyle Moon-Wright Hi, Please find the demo here: https://github.com/ralphwillgoss/prefect/tree/main/parallel Any questions let me know