Matthew Bell
04/22/2024, 9:41 PMMarvin
04/22/2024, 9:42 PMconcurrency
context manager, it can manage the number of concurrent tasks across all flows that are running. This ensures that, for instance, you do not exceed the maximum number of database connections or the number of parallel processing tasks regardless of how many flows are currently executing those tasks.
For example, by using concurrency("database", occupy=1)
, you can ensure that you never exceed the maximum number of database connections, or with concurrency("data-processing", occupy=5)
, you can limit the number of process_data
tasks to five at any one time across all running flows.
You can find more details on this feature in the Prefect documentation here: Global Concurrency Limits.