To limit the number of simultaneously submitted tasks in Prefect, you can use task concurrency limits which utilize task tags. Each tag can have an optional concurrency limit, representing the maximum number of concurrent task runs in a
Running state for tasks with that tag. If a task has multiple tags, it will run only if all tags have available concurrency. Note that tags without explicit limits are considered to have unlimited concurrency. Also, be cautious when using concurrency limits on tasks within subflows, as it can cause deadlocks.
For more details on how to configure task concurrency limits, refer to the following documents:
-
Task Concurrency Limits
-
Global Concurrency Limits
Remember, while configuring task run concurrency limits, set them wisely to prevent unwanted task run aborts when the limit is set to zero.