hello community, hello prefect team,
we were recently developing a new flow with very granular tasks which eventually resulted in a flow with 50k+ task runs. since we have no blocking statements like wait or result in our flow, this causes the flow to submit all these tasks at the very beginning. Since the PrefectClient has a connection pool of only 16 connections and a pool timeout of 30s, this led to a large number of pool timeout errors. increasing the timeout could prevent the timeout errors but submission of tasks still took very long and it seemed like the agent was only occupied submitting tasks and started a way lower number of tasks at a time than expected.
We eventually refactored our flow to use less granular tasks and parallelize over multiple flow runs using a flow of deployments in order to decrease the number of tasks per flow.
So i was wondering if you have any recommendations or experience with large number of tasks. how much can we expect prefect to handle? what's a sustainable task submission rate that wouldn't overwhelm the prefect client and are there any special best practices for massive flows?
Thanks a lot.