What's the recommended way to avoid reaching task submission rate limits documented at https://docs.prefect.io/latest/cloud/rate-limits/#flow-flow-run-and-task-run-rate-limits ? I tried setting Task Limits but that seems to limit the number of tasks executed concurrently rather than submission (which is what hits the cloud limits). For context, I have a data backfill flow that will spawn around 10,000 tasks (very quickly). I could batch up submission in code (and wait for the results before submitting more tasks), but I wondered if there's a more idiomatic way to do this which I've missed.

Hi @James Morley perhaps using our general-purpose concurrency limit API might fit the bill? The first example there includes a version where the concurrency limit is tasking inside a task, but it seems like you could adapt this to be inside the flow and around the

task.submit

call. Would this work?

hmm, I don't quite understand - the flow is submitting tasks (and getting back a

PrefectFuture

) sequentially (rather than in parallel), so limiting the concurrency of submitting tasks won't achieve anything?

Oh sorry, I meant to link to this section: https://docs.prefect.io/latest/guides/global-concurrency-limits/#using-rate_limit our global concurrency limit API functions as both concurrency AND rate limiting

nice, this is exactly what I need, thanks!

The idea is that you place a named rate limit in your flow code here, then you can go to the UI or use the API to set the limit on the fly