Is there a way to set a flow concurrency limit on the agent itself? As far as I can work out, work queue limits aren't going to work when we have multiple agents per queue, and agents that listen to multiple queues at the same time (with

agent start --match

). All I need to tell each agent is "Don't start more than N docker containers at the same time" (or you will run out of disk or memory)... E.g. the limit is really on the agent resources. In the full cloud I don't care how many things run simultaneously - the more the merrier :D

yes! you would do that by setting a concurrency limit on the work queue and having one work queue per agent

But wouldn't that mean that each deployment will only be executed on one agent - defeating the whole idea of having multiple agents to share the load ? (Sorry if I just misunderstood the relationship between work queues and agents)

gotcha, what you're describing is what we released in 2.6 with dynamic work queue pattern matching allowing you to essentially have one agent but per-deployment concurrency setting (run no more than 3 containers of this deployment on the agent dev) https://medium.com/the-prefect-blog/prefect-2-6-adds-configurable-results-dynamic-work-queue-patterns-and-custom-failure-handling-209cff4f71b6#1555

Awesome! Will go read - and thanks for the quick reply :)

I think you accidentally sent the last message to the channel, let's keep convo in the thread to avoid noise what do you mean by still per deployment? concurrency is set on a work queue and this utility of dynamic pattern matching makes it easier by creating multiple work queues (one per deployment if needed) while still leveraging one agent process

(oops, must accidently have click the small button in this text box - sorry) I need to limit the amount of flows running on an agent to avoid the agent running out of resources. Lets say to 4 simultaneous flow runs. It doesn't really matter what flows or what deployments - at least for now (but we do have some big flows that start quiet a few of tasks using map). I could do this by making one queue that goes into one agent and set the limit on it to 4. But as soon as I add a second agent, I'd have to increase this number to 8 - and does this mean 4 flow runs per agent? Besides, it also prevents me from having an agent listening to more than one queue at once, as it makes it difficult to control the limit on each agent. Currently, testing the setup with three agents: 1. Reads development flows from dev-staging and dev-testing queues. 2. Reads production flows from prod-noncritical and prod-critical queues. 3. Reads production flows from prod-critical queue only. Part of the setup is to ensure that critical production flows have high redundancy, and are not being starved of resources by other flows. At the same time the setup makes it easy to add agents moving forward. Since we're not running on a Kubernetes cluster or similar, multiple agents are currently our way of spreading the workload across multiple VMs - also with redundancy in mind.

Hi Thomas. It sounds like this issue might best be solved by spinning up more agents as needed based on resource needs using something like ECS with Fargate. That topic is discussed a bit in this thread. I think you could check the EC2 instance to see what it’s resource limits are and then spin up a new instance as needed, but that sounds like a bit of a bear. I’m not sure that agent concurrency limits are something we’re likely to do, almost certainly not in the next few months, but if that or something similar would be valuable, you’re welcome to open a feature request at the GitHub repo.

But as soon as I add a second agent, I’d have to increase this number to 8 - and does this mean 4 flow runs per agent?

If I understand your question correctly - if you had two agents pulling from the same work queue and had a concurrency limit of 8, it doesn’t guarantee that the split will be four flows for each agent - only that the combined total will be 8.

Hi @Jeff Hale, how about a pull request inside? Figured it was the least we could do 😉 Hope it is useful: https://github.com/PrefectHQ/prefect/pull/7361 And a bonus one for adding a tag to the flow run, while I'm at it: https://github.com/PrefectHQ/prefect/pull/7362 ... and thanks for the help!

Thank you, Thomas!

I am facing a similar issue. These 2 PRs would be a very nice addition for some of my projects as well. These can act as a workaround for a simple setup when a K8 solution isn't available and we want multiple agents to spread the workload across multiple local servers in a controlled way