https://prefect.io logo
a

Arjun Krishnamurthy

07/06/2023, 11:19 AM
Hi all, Can we have more than workers/agents polling a single work pool to distribute the work load or is it limited to have one agent/worker polling a work pool to avoid task duplication and inconsistent execution?
b

Bianca Hoch

07/06/2023, 4:57 PM
Hi Arjun, to my understanding, you can have multiple agents and workers polling a single work pool.
I believe the flow runs are distributed randomly across the workers in the scenario. To distribute the flow runs more evenly, you may consider adding a limit to the number of flow runs a worker or agent can initiate at a time.
For workers I believe this can be accomplished with the --limit option
a

Arjun Krishnamurthy

07/07/2023, 9:56 AM
@Bianca Hoch Got it. As per my understanding, limit can pickup multiple flow runs from more than one queue to run parallelly. Is that right.