I am trying to executed concurrent tasks of flow f...
# prefect-community
s
I am trying to executed concurrent tasks of flow from multiple agents with same work queue, but only single agent is picking the tasks. Is there a way to run these tasks on multiple agent with same queue? Using prefect - 2.6.4 #prefect-community #prefect-contributors
m
Hey @Shruti Hande Can you elaborate on your use case here a bit more? Agents are just responsible for submitting flow runs to the execution environment you have defined in the deployment for your flows, Multiple agents polling from the same work-queue may pull in the same flow runs but only one of them will actually execute it so it may just be a matter of timing. We did implement agent-level concurrency limits recently in prefect 2.7 which you could leverage to ensure a particular agent is only managing some pre-defined number of flow runs at any given time but but there isn't necessarily a way to force flow runs to split evenly between 2 agents polling for work from the same work-queue. These docs might be helpful as well for explaining this in more detail: • https://discourse.prefect.io/t/whats-the-role-of-agents-and-work-queues-and-how-the-concept-of-agents-differ-between-prefect-1-0-and-2-0/689https://docs.prefect.io/concepts/work-queues/
s
So multiple agents cannot distribute or split the task /flow runs from the same work queue?