@Marvin When using concurrent task runs with

.submit

is it possible to set a limit for the number of task runs that execute in parallell to not overload a flow with tasks?

Yes, it is possible to set a limit for the number of concurrent task runs when using

.submit

to prevent overloading the flow with tasks. This is accomplished through Prefect's built-in functionality for managing task concurrency limits, which uses task tags to govern the maximum number of concurrent task runs. Tasks can be throttled using the

rate_limit

function to ensure they are submitted at a controlled rate. Additionally, task concurrency limits can be applied based on tags, where you can specify the maximum number of concurrent task runs that are in a Running state. If a task has multiple tags, each tag's concurrency limit must be available for the task to run. Here is an example of using the

rate_limit

function to throttle task submission:

python
from prefect import flow, task
from prefect.concurrency.sync import rate_limit

@task
def my_task(i):
    return i

@flow
def my_flow():
    for _ in range(100):
        rate_limit("slow-my-flow", occupy=1)
        my_task.submit(1)

if __name__ == "__main__":
    my_flow()

And to apply a concurrency limit based on task tags, simply specify the limit when defining your tasks and tag them accordingly. For more information and a detailed guide, you can refer to the documentation on global concurrency limits and task concurrency limits.