https://prefect.io logo
j

Joni Pelham

08/23/2023, 2:58 PM
Has anyone else found mapped tasks can take a long time for prefect to organise before it starts processing them? https://docs.prefect.io/2.11.4/concepts/tasks/#pacc-may-31-2023 Works fine with low numbers of items to map over but beyond 100 it takes more than 20 minutes and sometimes doesn't finish. I have some 9000+ items I need to process.
1
I think i've worked this out. I had a hunch too much memory was being required(though i had no diagnostics). I refactored to reduce the amount being passed to each of the instances and it seems to have worked.
Issues now are how to scale things up. I adopted a results output strategy which appends but for large numbers of workers that causes conflicts as they all output to the same file. Also presumably if I use map with larger amounts of data I will run into the same issue.