Has anyone else found mapped tasks can take a long time for

Question

Has anyone else found mapped tasks can take a long time for prefect to organise before it starts processing them <https docs prefect io 2 11 4 concepts tasks pacc may 31 2023> Works fine with low numbers of items to map over but beyond 100 it takes more than 20 minutes and sometimes doesn t finish I have some 9000+ items I need to process

Joni Pelham · Answer

I think i ve worked this out I had a hunch too much memory was being required though i had no diagnostics I refactored to reduce the amount being passed to each of the instances and it seems to have worked

Joni Pelham · Answer

Issues now are how to scale things up I adopted a results output strategy which appends but for large numbers of workers that causes conflicts as they all output to the same file Also presumably if I use map with larger amounts of data I will run into the same issue