Thanks for the details — I can see why this is frustrating. A few important points about how concurrency limits work in Prefect 3.x with ECS workers and why you’re still hitting the AWS vCPU limit: High-level summary - Work pool/queue concurrency limits are best-effort and only limit how many runs a worker will pick up. They do not strictly prevent runs from starting; race conditions can still occur. - The most reliable way to cap concurrent ECS tasks is to use deployment-level concurrency limits and/or global (tag-based) concurrency limits which block the state transition to RUNNING. - Also, calling a flow function directly creates a subflow in the same process; calling a deployment (via

run_deployment

) creates a new ECS task. Which one you’re doing matters for how limits apply. A couple clarifying questions (so I can tailor the fix precisely): 1) Are you on Prefect 3.x? If you’re on 2.x, the guidance differs a bit and I’ll adjust. 2) How are you triggering

do_prediction

in practice? Your snippet shows a direct call

do_prediction(...)

inside a task, but your description sounds like you may actually be submitting deployment runs (e.g., via

run_deployment

) to an ECS work pool. 3) Which work queue is the deployment actually using? Is the concurrency limit set on that exact queue (not just the pool)? Why your current limits aren’t stopping AWS vCPU errors - Work queue (and pool) concurrency limits are soft controls during polling. They reduce the number of runs handed to workers but do not strictly block state transitions and can be subject to races under parallel submission. - If multiple submissions happen quickly, workers may still start more runs than your ECS quota allows. ECS will reject with “You’ve reached the limit on the number of vCPUs...”, and Prefect will surface that failure. - If you are calling

run_deployment(as_subflow=False)

(or submitting runs externally), each run will start a new ECS task — so you need a strict cap at the orchestration layer, not just queue polling. What to do (reliable fixes) Option A: Add a global (tag-based) concurrency limit — most robust - Tag your deployment/runs (e.g.,

ecs-prediction

) and create a global limit for that tag. This enforces limits at state transition time so runs won’t start if the limit is reached. - Add a tag to your flow/deployment: - In code when deploying:

flow.from_source("<https://github.com/org/repo>", entrypoint="path/to/file:do_prediction").deploy(
        name="predictions",
        work_pool_name="name-pool",
        tags=["ecs-prediction"]
    )

- Or tag via the UI/CLI when creating the deployment. - Create the global limit (verified CLI):

prefect global-concurrency-limit create "tag:ecs-prediction" --limit 6

Docs: - Global concurrency limits - Tag-based concurrency limits - How-to: Apply global limits and Tag-based limits Option B: Set a deployment-level concurrency limit - This caps concurrent runs of a specific deployment. - When you deploy:

flow.from_source("<https://github.com/org/repo>", entrypoint="path/to/file:do_prediction").deploy(
      name="predictions",
      work_pool_name="name-pool",
      concurrency_limit=6,
      tags=["ecs-prediction"]
  )

Docs: - Deployments and Create deployments These options stack. The most reliable pattern is: - Use a global limit (tag-based) to cap total ECS tasks across deployments. - Use per-deployment concurrency if you want an additional per-deployment cap.

- Keep queue concurrency as a safety net, but don’t rely on it for strict control. Triggering correctly from your task - If you want each prediction to run in its own ECS task and be governed by the limits above, submit deployment runs and do not call the flow function directly. Use

run_deployment

and set

as_subflow=False

to ensure it runs as a separate flow run on your ECS pool:

from prefect import task
  from prefect.deployments import run_deployment

  @task
  async def trigger_parallel_prediction(tile_name: str):
      # Submit a separate ECS task (flow run) for this tile
      return await run_deployment(
          name="do_prediction/predictions",  # "flow_name/deployment_name"
          parameters={"tile_name": tile_name},
          as_subflow=False  # ensures a separate infrastructure run
      )

Signature (verified):

run_deployment(name, parameters=None, work_queue_name=None, as_subflow=True, ...)

- If you omit

as_subflow=False

, calling from within another flow will default to a subflow (runs in the same process) and won’t create a new ECS task. - Docs: Run deployments ECS worker notes - Each deployment run picked up by an ECS worker starts a new ECS task; to avoid AWS quota errors you must prevent runs from entering RUNNING beyond your quota. That’s exactly what global/deployment concurrency limits do. - ECS worker docs: ECS worker - Serverless/push work pools: Serverless push pools One more important note - In Prefect 3.x there is no

@ecs

decorator. Infrastructure is chosen by the deployment’s work pool, not via a decorator on the flow. If you’re on 3.x, you’ll want to deploy with

flow.from_source(...).deploy(work_pool_name="name-pool", ...)

and then trigger with

run_deployment

. If you’re on 2.x or using a custom decorator, let me know and I’ll tailor the steps. If you can share: - Prefect version - How you’re submitting runs (direct flow call vs

run_deployment

, and whether

as_subflow=False

) - Which work queue the deployment targets …I can provide an exact set of commands and code for your setup.

@Marvin Adding a global (tag-based) concurrency limit did not work, as the active flows are detected, but they count as active for only 30 seconds. is it possible to remove the time limit and just wait for them to finish?