We are having some very highly mapped jobs (around 10 jobs each having 200,000 mapped tasks) which is causing strain for the prefect server on task status update making apollo unavailable. Any recommendation to reduce the strain (aside from auto-scaling of prefect server)?
12/16/2021, 1:47 PM
In general, you could consider migrating to Prefect Cloud which was built for such scale, as this answer shows.
If you need a quicker solution, perhaps you can split this into separate flows so that each can handle a smaller amount of tasks?