Feature idea: Looks like there isn’t a way to spr...
# ask-community
d
Feature idea: Looks like there isn’t a way to spread the work load between many prefect agents: https://discourse.prefect.io/t/agents-and-work-queues-faq/1468#what-are-use-cases-where-starting-multipl[…]r-the-same-work-queue-can-be-beneficial-4 Currently I have many agents working on the same queue and it happens that 1 agent fetches many flow runs and unbalances the work load, threatening sometimes to use more resources than it’s host machine has. My current solution for this is using the
--limit
flag in the agent, but that’s not ideal. I though it would be nice to have a
--prefetch-limit
option to the agent (beyond the already present
--prefetch-seconds
). This way, an agent could have a big
--limit
but not fetch too many flow runs at once, therefore spreading a bit the load to the other agents. What do you think?
r
This is where k8s comes in and the agents simply create jobs that get run across pods/nodes
z
That’d just reduce the number of runs grabbed per agent interval
d
Thank you for the responses! @redsquare I see the value of that, but then I would need to integrate the k8 infrastructure with my code. I think @Zanie suggestion would already be a good enough solution for me 🙂
👍 1
if you want I can create an issue for this in github.
z
I’d appreciate that!
d