Feature idea: Looks like there isn’t a way to spread the work load between many prefect agents: https://discourse.prefect.io/t/agents-and-work-queues-faq/1468#what-are-use-cases-where-starting-multipl[…]r-the-same-work-queue-can-be-beneficial-4 Currently I have many agents working on the same queue and it happens that 1 agent fetches many flow runs and unbalances the work load, threatening sometimes to use more resources than it’s host machine has. My current solution for this is using the

--limit

flag in the agent, but that’s not ideal. I though it would be nice to have a

--prefetch-limit

option to the agent (beyond the already present

--prefetch-seconds

). This way, an agent could have a big

--limit

but not fetch too many flow runs at once, therefore spreading a bit the load to the other agents. What do you think?

This is where k8s comes in and the agents simply create jobs that get run across pods/nodes

We could make the

limit

at https://github.com/PrefectHQ/prefect/blob/main/src/prefect/agent.py#L214 be a setting

Thank you for the responses! @redsquare I see the value of that, but then I would need to integrate the k8 infrastructure with my code. I think @Zanie suggestion would already be a good enough solution for me 🙂

I’d appreciate that!

Here it is! https://github.com/PrefectHQ/prefect/issues/8650