Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Has anyone else found mapped tasks can take a long time for prefect to organise before it starts processing them? <https://docs.prefect.io/2.11.4/concepts/tasks/#pacc-may-31-2023>
Works fine with low numbers of items to map over but beyond 100 it takes more than 20 minutes and sometimes doesn't finish.  I have some 9000+ items I need to process.

I think i've worked this out.  I had a hunch too much memory was being required(though i had no diagnostics).
I refactored to reduce the amount being passed to each of the instances and it seems to have worked.

Issues now are how to scale things up.  I adopted a results output strategy which appends but for large numbers of workers that causes conflicts as they all output to the same file.
Also presumably if I use map with larger amounts of data I will run into the same issue.