hello community, hello prefect team, we were recently developing a new flow with very granular tasks which eventually resulted in a flow with 50k+ task runs. since we have no blocking statements like wait or result in our flow, this causes the flow to submit all these tasks at the very beginning. Since the PrefectClient has a connection pool of only 16 connections and a pool timeout of 30s, this led to a large number of pool timeout errors. increasing the timeout could prevent the timeout errors but submission of tasks still took very long and it seemed like the agent was only occupied submitting tasks and started a way lower number of tasks at a time than expected. We eventually refactored our flow to use less granular tasks and parallelize over multiple flow runs using a flow of deployments in order to decrease the number of tasks per flow. So i was wondering if you have any recommendations or experience with large number of tasks. how much can we expect prefect to handle? what's a sustainable task submission rate that wouldn't overwhelm the prefect client and are there any special best practices for massive flows? Thanks a lot.

following

We’re definitely thinking about this but I’m not sure I have concrete recommendations yet.

hey Zanie, we are already using the ray task runner and made use of PREFECT_API_REQUEST_TIMEOUT but saw that things would get stuck at some point which was when we decided to refactor for less tasks. running on prefect 2.9.0 and the prefect cloud. Cloud you please elaborate how jitter would solve the submission problem?

The problem (I’m looking into solving with jitter) is all of the task state transitions from PENDING -> RUNNING get queued after all of the task run creation requests which means all of the task runs must be created before any of them start running. Adding jitter means that some of the state transitions get proposed and the tasks can start running while creation continues in the background.

following as well. I was looking for what I was doing wrong with spawning a lot of granular tasks. I had kind of assumed they would go into a queue and get picked off by workers, but I think my mental model was incorrect lol