<@ULVA73B9P> i have a work queue with concurrency ...
# ask-marvin
a
@Marvin i have a work queue with concurrency 6 but only 1 flow can run at a time, whilst any other flows scheduled at the same time gets pushed to late.why is this?
m
thought for 379.8 seconds
Short answer: a work queue concurrency of 6 only controls how many runs the server will hand out. You still need enough worker capacity to actually start multiple runs. If your worker is effectively limited to 1, everything else scheduled at the same time will sit Pending and quickly become Late. Common causes to check (assuming Prefect 3.x): - Worker capacity set to 1 - A single worker can run multiple flow runs in parallel only if its own limit allows it. If you started your worker without increasing its limit (or with a limit of 1), you’ll see exactly this behavior. - Action: start your worker with enough capacity or run multiple workers. - Docs: Workers concept - Work pool concurrency limit lower than queue limit - If the work pool is limited to 1, that wins over the queue limit. - Check it:
Copy code
prefect work-pool inspect <POOL>
- Adjust it:
Copy code
prefect work-pool set-concurrency-limit <POOL> 6
- Docs: Work pool CLI - Global concurrency limits (tag-based) restricting to 1 - If your runs have a tag that’s governed by a global concurrency limit of 1, only one run can start at a time. - Check:
Copy code
prefect gcl ls
- Docs: Global concurrency limits - Schedule/deployment constraints - If your schedule or deployment configuration limits active runs (or requires approval), additional runs won’t start and will become Late. - Docs: Schedules - Infrastructure limits - For Docker/Kubernetes pools, cluster/node capacity or the base job template can effectively limit you to one run. Check worker logs for repeated submission failures or backoffs. Helpful commands to diagnose: - Inspect queue (see concurrency and state):
Copy code
prefect work-queue inspect -p <POOL> <QUEUE>
- Inspect pool:
Copy code
prefect work-pool inspect <POOL>
- List global concurrency limits:
Copy code
prefect gcl ls
If you can share: - How you’re starting the worker (command and options) - Output of
prefect work-queue inspect -p <POOL> <QUEUE>
and
prefect work-pool inspect <POOL>
- Whether you have any global concurrency limits (
prefect gcl ls
) - The work pool type (process, docker, kubernetes) I can pinpoint exactly which limit is holding you to a single concurrent run and recommend the precise fix.
a
@Marvin how do i check Worker capacity
m
thought for 94.4 seconds
Great question. In Prefect 3.x, a worker’s “capacity” is the max number of flow runs it will start concurrently (its limit). Here’s how to check and reason about it: What to look at - The worker’s own limit (per-process capacity) - If you started the worker with a limit (e.g.,
--limit 6
), that is its capacity. - If you did not set a limit, the worker itself does not enforce one (None), but your effective capacity can still be 1 if other limits exist (pool/queue/global). - Pool/queue/global limits (these can cap your worker even if the worker’s limit is high) - Work pool concurrency - Work queue concurrency - Global concurrency limits (by tag) How to check - Worker process (your machine/container logs) - Check the command used to start the worker; if you included a limit, that’s the capacity:
Copy code
prefect worker start -p <POOL> -q <QUEUE> --limit 6
- If you manage the worker via systemd/Docker/K8s, check the startup command/env in those manifests. The worker typically logs its config (including limit) on startup. - Prefect UI - Go to Work Pools > select your pool > Workers tab. You can see your worker, its status/health, and the number of active runs it’s handling. This helps you infer whether it’s starting more than one at a time. - CLI for pool/queue/global limits (these affect effective capacity): - Work pool concurrency:
Copy code
prefect work-pool inspect <POOL>
Adjust if needed:
Copy code
prefect work-pool set-concurrency-limit <POOL> 6
Docs: Work pool CLI - Work queue concurrency:
Copy code
prefect work-queue inspect -p <POOL> <QUEUE>
Docs: Work queue CLI - Global concurrency limits (by tag):
Copy code
prefect gcl ls
Docs: Global concurrency limits If you share how you’re starting the worker (the exact
prefect worker start ...
command) and the outputs of
work-pool inspect
,
work-queue inspect
, and
prefect gcl ls
, I can tell you precisely what your current effective capacity is and what’s capping it.