Joni Pelham

08/23/2023, 2:58 PM
Has anyone else found mapped tasks can take a long time for prefect to organise before it starts processing them? Works fine with low numbers of items to map over but beyond 100 it takes more than 20 minutes and sometimes doesn't finish. I have some 9000+ items I need to process.
I think i've worked this out. I had a hunch too much memory was being required(though i had no diagnostics). I refactored to reduce the amount being passed to each of the instances and it seems to have worked.
Issues now are how to scale things up. I adopted a results output strategy which appends but for large numbers of workers that causes conflicts as they all output to the same file. Also presumably if I use map with larger amounts of data I will run into the same issue.