Does anyone know how global concurrency limits int...
# ask-community
d
Does anyone know how global concurrency limits interact with task-level concurrency limits? i.e. if I have a gcl called "foo" and a task-level concurrency limit with the tag "foo", does one of them take priority and the other is ignored? If so, which, and if not, what happens? @Marvin do you know?
m
It seems you've stumbled upon the intricacies of balancing global and task-level concurrency limits in Prefect. To clarify, both global concurrency limits and task-level limits can be set in Prefect, but they serve slightly different purposes: 1. Global Concurrency Limits: These are applied more broadly across your Prefect environment. They control how many concurrent operations (tasks, flows, etc.) can run simultaneously. These limits are useful for optimizing resource usage and preventing system bottlenecks. Global limits are set if your primary goal is resource management across your entire environment. 2. Task-Level Concurrency Limits: Set at the task level, these limits control how many instances of a particular task can run at the same time. This is useful when you have tasks that may conflict with one another if run concurrently or when you want to optimize the behavior of specific tasks. If you've set both a global concurrency limit and a task-level concurrency limit using the same tag (like "foo"), neither limit is ignored. Instead, they work together. The global limit applies to all operations sharing the tag across your Prefect environment. Meanwhile, the task-level limit specifically manages concurrency for instances of that tagged task. Thus, the task-level limit acts as a more granular control within the broader scope of the global limit. For further reading, you can explore Prefect's documentation on global concurrency limits here. And remember, even in a world of advanced task orchestration, it's always wise to mind the limits—or they'll certainly mind you.