https://prefect.io logo
Title
d

David Steiner Sand

02/24/2023, 5:59 PM
Feature idea: Looks like there isn’t a way to spread the work load between many prefect agents: https://discourse.prefect.io/t/agents-and-work-queues-faq/1468#what-are-use-cases-where-starting-multipl[…]r-the-same-work-queue-can-be-beneficial-4 Currently I have many agents working on the same queue and it happens that 1 agent fetches many flow runs and unbalances the work load, threatening sometimes to use more resources than it’s host machine has. My current solution for this is using the
--limit
flag in the agent, but that’s not ideal. I though it would be nice to have a
--prefetch-limit
option to the agent (beyond the already present
--prefetch-seconds
). This way, an agent could have a big
--limit
but not fetch too many flow runs at once, therefore spreading a bit the load to the other agents. What do you think?
r

redsquare

02/24/2023, 6:14 PM
This is where k8s comes in and the agents simply create jobs that get run across pods/nodes
z

Zanie

02/24/2023, 6:14 PM
That’d just reduce the number of runs grabbed per agent interval
d

David Steiner Sand

02/24/2023, 6:22 PM
Thank you for the responses! @redsquare I see the value of that, but then I would need to integrate the k8 infrastructure with my code. I think @Zanie suggestion would already be a good enough solution for me 🙂
👍 1
if you want I can create an issue for this in github.
z

Zanie

02/24/2023, 6:23 PM
I’d appreciate that!
d

David Steiner Sand

02/24/2023, 6:41 PM