Fugo Takefusa
09/24/2025, 9:55 PMMarvin
09/24/2025, 9:55 PMMarvin
09/24/2025, 10:05 PMprefect worker start ...
command)?
- Did you set a concurrency limit on the deployment (Max concurrent runs)?
Checklist to enable concurrent runs
1) Check the deployment’s own concurrency limit
- A deployment-level limit of 1 will serialize runs regardless of pool settings.
- Inspect it:
prefect deployment ls
prefect deployment inspect "<flow-name>/<deployment-name>"
Look for concurrency_limit
(should be None/Unlimited).
- If you deployed in code, ensure you didn’t pass concurrency_limit=1
to flow.deploy(...)
.
Docs: Deployments
2) Check for tag-based global concurrency limits
- If your flow or tasks have tags that are limited, runs will queue.
prefect concurrency-limit ls
If you see a limit on a tag your deployment uses, either delete it or raise it:
prefect concurrency-limit delete <tag>
Docs: Global concurrency limits
3) Confirm the worker isn’t limiting you to 1
- If your worker was started with a concurrency cap (e.g., --limit 1
), it will only start one flow at a time. Check the command you used to start it and remove/increase any limit.
- You can also run multiple workers against the same pool to scale out:
prefect worker start -p <your-ecs-pool> -n worker-1
prefect worker start -p <your-ecs-pool> -n worker-2
Docs: Workers, Work pools
4) Verify the work pool is truly unlimited
- You mentioned it’s unlimited — good! Just to double-check:
prefect work-pool inspect "<your-ecs-pool>"
Ensure no concurrency limit is set.
5) Confirm the ECS side isn’t the bottleneck
- The Prefect ECS worker submits a separate RunTask per flow run. If the second run is stuck in “AwaitingInfrastructure”, check worker logs for AWS errors (e.g., Fargate vCPU quota, subnets without IPs, etc.). If AWS capacity is the issue, you’ll see ECS errors in the worker logs and need to raise quotas or adjust networking.
- If you’re using a custom job template that creates an ECS Service with desiredCount: 1
, that will also serialize work; use RunTask (the default) instead.
Docs: ECS worker guide
If you can share:
- The run state of the “waiting” flow
- The output of prefect deployment inspect "<flow>/<deployment>"
- The output of prefect concurrency-limit ls
- The exact worker start command and any worker logs around the time the second run is queued
…I can pinpoint the exact gate and give you the minimal change to enable parallel runs.