<@ULVA73B9P> i switched from `ecs:push` to `ecs` ...
# ask-marvin
l
@Marvin i switched from
ecs:push
to
ecs
worker pools and now my global concurrency limit in prefect cloud is being ignored and I am running out of vcpu. Is this a known issue? do i need to recreate the global concurrency limit since i made a new workpool or is this a coincidence it broke after this change?
m
thought for 509.1 seconds
Short answer: switching from an ecs:push work pool to ecs should not affect Global Concurrency Limits (GCLs). GCLs are enforced by Prefect Cloud’s orchestration layer and are independent of the worker pool type. You should not need to recreate a GCL just because you made a new work pool. This isn’t a known issue with ecs pools. Most common reasons a “global limit” appears to be ignored after a pool change: - You were actually relying on a work-pool concurrency limit on the old pool and didn’t set it on the new pool. Work-pool concurrency is separate from Global Concurrency Limits. - The GCL exists but your flow runs are not acquiring it (GCLs only apply when acquired by name in code). - The GCL is disabled or misnamed. Quick checks 1) Verify your GCL is present and enabled
Copy code
prefect global-concurrency-limit ls
prefect global-concurrency-limit inspect <limit-name>
prefect global-concurrency-limit enable <limit-name>
Docs: Global concurrency limits 2) Confirm your code is acquiring the GCL - Sync:
Copy code
from prefect.concurrency.sync import concurrency

with concurrency("<limit-name>"):
    # your work
- Async:
Copy code
from prefect.concurrency.asyncio import concurrency

async with concurrency("<limit-name>"):
    # your work
If your flows don’t acquire the limit by name, the GCL won’t throttle anything. 3) Set a concurrency limit on the new work pool (if you were using pool-level limits before)
Copy code
prefect work-pool set-concurrency-limit "<your-ecs-pool-name>" 10
CLI ref: Work pool CLI 4) (Optional) If you were relying on deployment-level concurrency, double-check that setting on your deployment after the migration. Not a known bug - We’re not tracking an issue where ecs pools ignore GCLs. The only related reports we’ve seen are about scheduling behavior on ecs:push, not GCL enforcement. If you can share: - The GCL name you expect to be enforced and the output of
prefect global-concurrency-limit inspect <name>
- Whether your flow code acquires that GCL - Whether you’ve set a concurrency limit on the new ecs work pool …I can help pinpoint which of the above is causing the spike in vCPU.
l
makes sense - i have set a limit on both the work pool and the default queue. do tasks get held as "late" when the queue is full?
@Marvin do tasks get held as "late" when the queue is full?
m
thought for 95.1 seconds
Great question — short answer: it depends what you mean by “queue.” - Work pool/queue full: This only affects flow runs. If your work pool (or a queue within it) is at capacity and a flow run’s scheduled start time passes before a worker picks it up, that flow run will show as Late. Tasks are not placed in work pool queues. - Tasks inside a flow: Task runs can also show Late, but not because the work pool queue is full. Tasks become Late when their own scheduled start time passes and the engine hasn’t started them yet — typically due to the task runner/execution engine being saturated (e.g., limited threads/workers, Dask/Ray cluster at capacity) or other in-flow concurrency constraints. If you’re seeing “Late,” it’s worth confirming whether it’s on the flow run (work pool capacity issue) or on specific task runs (task runner capacity issue). References: - States: Scheduled vs Late - Work pools and queues If you can share whether the “Late” label appears on flow runs or task runs (and what task runner you’re using), I can suggest concrete steps to address it.