I am trying to executed concurrent tasks of flow from multiple agents with same work queue,
but only single agent is picking the tasks.
Is there a way to run these tasks on multiple agent with same queue?
Using prefect - 2.6.4
#CL09KU1K7#C0106HZ1CMS
m
Mason Menges
12/09/2022, 4:57 PM
Hey @Shruti Hande Can you elaborate on your use case here a bit more? Agents are just responsible for submitting flow runs to the execution environment you have defined in the deployment for your flows, Multiple agents polling from the same work-queue may pull in the same flow runs but only one of them will actually execute it so it may just be a matter of timing. We did implement agent-level concurrency limits recently in prefect 2.7 which you could leverage to ensure a particular agent is only managing some pre-defined number of flow runs at any given time but but there isn't necessarily a way to force flow runs to split evenly between 2 agents polling for work from the same work-queue.
These docs might be helpful as well for explaining this in more detail:
• https://discourse.prefect.io/t/whats-the-role-of-agents-and-work-queues-and-how-the-concept-of-agents-differ-between-prefect-1-0-and-2-0/689
• https://docs.prefect.io/concepts/work-queues/
s
Shruti Hande
12/13/2022, 8:50 AM
So multiple agents cannot distribute or split the task /flow runs from the same work queue?