<@ULVA73B9P> a pod is running for a deployment flo...
# ask-marvin
s
@Marvin a pod is running for a deployment flow run and another deployment flow run is scheduled to start every 5 minutes. If the current flow run hasn't completed can Prefect stop the next pod from starting based on a work queue concurrency setting?
m
thought for 306.4 seconds
Short answer: yes. In Prefect 3.x you can prevent overlapping flow runs by setting a concurrency limit on the work queue (or the work pool) that your deployment uses. If you set the queue’s concurrency limit to 1, Prefect will hold subsequent scheduled runs in a queued/awaiting-concurrency state and will not start another pod until the currently running flow run finishes. Key points - Work queue concurrency applies to all deployments using that queue. If you only want to limit a single deployment, give it a dedicated queue and set the limit there. - Pool-level concurrency applies to all queues/work in the pool. - This gating happens in Prefect’s orchestration layer, so a Kubernetes pod won’t be created until a concurrency slot is available. CLI snippets - Set a limit on an existing work queue:
Copy code
prefect work-queue set-concurrency-limit <queue-name> 1 --pool <pool-name>
- Create a queue with a limit:
Copy code
prefect work-queue create <queue-name> --pool <pool-name> --limit 1
- Pool-wide limit (optional, broader scope):
Copy code
prefect work-pool set-concurrency-limit <pool-name> 1
Docs for more detail: - CLI: work-queue - CLI: work-pool - How-to: concurrency limits Questions to tailor the setup: - Are you on Prefect Cloud or OSS? - Is this deployment sharing a work queue with other deployments, or can you give it a dedicated queue?
s
@Marvin, does this apply to Prefect v2.x?
m
thought for 51.5 seconds
Yes—this works in Prefect 2.x as well. Ways to prevent overlap in 2.x: - Work queue or work pool concurrency limit: Set the queue (or pool) limit to 1. The next scheduled run will stay queued/awaiting concurrency and the agent/worker will not start another Kubernetes pod until the current run finishes. For a per-deployment effect, give the deployment its own queue and set that queue’s limit to 1. - Tag-based concurrency limit: Add a unique tag to the deployment/flow and set a concurrency limit of 1 on that tag. Any flow runs with that tag will not overlap. Notes - If you share a queue and set the limit to 1, you’ll throttle all deployments on that queue. Use a dedicated queue if you only want to gate a single deployment. - In early 2.x you’ll be using agents; in later 2.x you may be using workers/work pools. The behavior is the same: Prefect won’t submit another job/pod until a slot is free. If you can share your exact 2.x version and whether you’re using agents or workers, I can give precise steps. Also, if you’re open to it, upgrading to Prefect 3.x gives you the same controls with some UX improvements.