I have multiple deployments in a work queue with u...
# prefect-community
a
I have multiple deployments in a work queue with unlimited concurrency, but they still run sequentially and one will not start until the other is finished. Am I misunderstanding how concurrency works and it only applies to the same flow? Do I need a separate work queue for each of my deployments? Because what has been happening is we occasionally have a flow that takes a really long time, and all other flows wait until the long-running one finishes, with any missed jobs showing up as late and then running once the long one finishes. Two issues with that: having time-sensitive jobs not run because they are waiting, and then having hundreds of late jobs in the queue and no way to get rid of them other than deleting one by one
m
Sorry noobie answering here, but isn't there a concurrecy configuration that needs to be set?
a
There is, and we had this issue with concurrency set to 5. Then I created a new work queue with unlimited concurrency and two deployments for testing, and the second deployment still waited for the other one to complete before running. It seems like concurrency only applies to an individual flow, hoping to get confirmation
m
The short answer here is yes the concurrency is restricting concurrent runs of matching flows. https://docs.prefect.io/concepts/work-queues/#work-queue-concurrency Outside of that The Agent will continue to pull for work from the workqueue and submit the jobs provided there are resources available in the execution environment available to run the flows, if that's not the case they can get stuck in pending. you can divide the work across multiple agents/workqueues depending on how you want to organize everything
👍 1
a
Thank you for the confirmation, I will use separate work queues