Great question — and nice job pulling the logs. Short answer: a 20-minute gap between “submitting flow run” and “Creating ECS task run” almost always means the run was waiting in Prefect’s queue due to concurrency/queueing limits on the Prefect side, not an ECS capacity issue. The key clue: your worker didn’t call ECS until 06:48 (that’s when it first logs “Retrieving ECS task definition…”). Everything before that was Prefect orchestration, not AWS. Most common causes of this gap - Work pool concurrency limit reached: If your pool has a concurrency cap, new runs wait until a slot opens. - Worker capacity saturated: A single worker (or low worker concurrency) handling many runs can delay submissions. - Flow/tag concurrency limits: Concurrency keys or tag-based limits can hold a run until a slot opens. - Run queue backlog: Lots of pending runs relative to worker count and concurrency. - AWS API throttling before RunTask: If the worker was being throttled by AWS, it will backoff/retry (you’d usually see throttling messages in logs). - Worker availability: If the worker briefly went offline or was overloaded, it may have rescheduled the submission. How to confirm what happened - In the Prefect UI, open the flow run Timeline and State Details: - Look for time spent in “Scheduled/Not Ready/Waiting” states and any “reason” (e.g., concurrency limit reached). - If it jumped straight to “Submitting” and then nothing for 20 min, the worker likely rescheduled the submission due to limits/backoff. - On the Work Pool page: - Check the pool’s Concurrency limit and current Active runs. - Check Run queue length and “queue latency”. - Confirm your workers are healthy and how many are connected. - Worker logs at DEBUG: - Set

PREFECT_LOGGING_LEVEL=DEBUG

when starting the worker to see messages like concurrency wait/backoff. - Cross-check in AWS: - In CloudTrail, filter for

ecs:RunTask

to see the exact time the API was called. It should align with ~06:48, confirming the delay was pre-AWS. What to do if this is the cause - Increase the work pool’s concurrency limit if it’s undersized for your workload. - Add more workers or increase a worker’s concurrency (run more worker processes/containers/EC2s). - Distribute runs across multiple work pools/work queues to isolate heavy workloads. - Revisit flow/tag concurrency limits if they’re too restrictive. - If you see AWS throttling, consider staggering submissions, adding more workers in more regions, or requesting a quota increase. References - ECS worker guide (Prefect 3.x) - Work pools and concurrency (Prefect 3.x) - Worker health checks (Prefect 3.x) If you can share: - The work pool name and its concurrency limit - How many workers you’re running for that pool and how they’re started - Whether you use any flow/tag concurrency limits - A screenshot of the flow run Timeline/State details around 0627–0648 …I can help pinpoint which limit caused the wait and suggest the right tuning.

Short answer: based on those timestamps, the wait happened before ECS. The RunTask call isn’t made until a Prefect ECS worker picks up the flow run and starts provisioning infra. Since CloudTrail shows RunTask at 6:48 and your first ECS-related worker log (“retrieving ECS task definition”) is also at 6:48, the gap from 6:27 → 6:48 was between the run being submitted and the worker picking it up. A few things to check to pinpoint why the worker didn’t pick it up sooner: - Sanity-check timezones - CloudTrail is UTC; Prefect UI/logs are in your browser’s local time. Make sure 6:27 vs 6:48 aren’t just a TZ mismatch. - Flow run state timeline - Open the flow run page and look at the state history. If the run’s Scheduled time was 6:48, the behavior is expected. - If it was Scheduled immediately at ~6:27 but only moved to “Infrastructure starting” at ~6:48, keep checking the items below. - Worker availability and polling - In the UI: Work Pools > your ECS pool. Confirm the pool wasn’t paused and that a worker was online during 627–648. The pool/worker pages show “last polled” timestamps. - Check your worker logs for that window. You should see regular “polling for work” messages; if those aren’t present, the worker may have been offline or pointed at the wrong queue/tags. - Work queue filters / tags mismatch - Ensure the deployment’s work queue name and any tags match what the worker is pulling from (via

--work-queue

and tags). - If there’s a mismatch, the run will sit Scheduled until a matching worker appears. - Concurrency limits - Pool-level concurrency: make sure the work pool doesn’t have a restrictive concurrency limit that was saturated. - Tag-based limits: check if your flow or tasks use tags that have active concurrency limits and were at capacity. - CLI to inspect:

prefect work-pool inspect "<your-pool>"
    prefect concurrency-limit ls

You can also list runs to see if others were occupying slots:

prefect flow-run ls --state Running,Scheduled --limit 50

- Worker logs around pickup - When the worker actually picks up the job, you’ll see logs like “Submitting infrastructure” → “Retrieving ECS task definition” → “RunTask”. If those only appear at 6:48, the worker did not see (or could not accept) the run before then. If you can share: - Prefect version (assuming 3.x unless you tell me otherwise) - Flow run ID - Work pool name and whether it has a concurrency limit - The worker command you used to start it (including any `--work-queue`/tags) - A snippet of worker logs from 627–648 …I can help pinpoint the exact cause. Docs for reference: - ECS guide (Prefect 3.x) - Work pools (Prefect 3.x) - Settings reference

Short answer: yes, a single worker can become a bottleneck when you trigger a large number of flow runs. You don’t need priorities to scale—just add more workers and make sure your pool isn’t constrained by a concurrency limit. What to do 1) Check and adjust your work pool’s concurrency limit - If your pool has a low limit, runs will queue regardless of worker count.

prefect work-pool inspect <your-pool>
prefect work-pool set-concurrency-limit <your-pool> <new_limit>

Docs: prefect work-pool CLI 2) Start additional workers for the same pool - You can run multiple worker processes on the same machine or on different machines/containers. Each worker polls the pool and launches infrastructure for runs.

# start another worker (name is optional)
prefect worker start -p <your-pool> --name worker-2

# optionally shard by queues if you use them
prefect worker start -p <your-pool> -q <queue-A> --name worker-a
prefect worker start -p <your-pool> -q <queue-B> --name worker-b

Docs: Workers concept Notes and gotchas - Per-worker concurrency: there isn’t a hard per-worker concurrency limit in Prefect; workers submit runs to your infrastructure. Throughput is governed by your pool’s concurrency limit and your infra capacity (e.g., how many Docker/K8s/ECS jobs can run at once). - Task-level throttling: if you’ve created tag-based concurrency limits, those can also cap throughput:

prefect concurrency-limit ls

Docs: Concurrency limits CLI - Priorities are optional; you can ignore them if you just want more throughput. A couple of quick questions so I can tailor this: - What type of work pool are you using (Process, Docker, Kubernetes, ECS, etc.)? - What symptom are you seeing—runs stuck in Scheduled for a long time, or just slower start-up? - Do you already have any pool-level or tag-based concurrency limits configured?

Short answer: No. Setting

PREFECT_LOGGING_LEVEL=DEBUG

will make the process you set it on more verbose, but worker logs do not show up in a flow run’s logs. Flow run logs and worker logs are separate. How to use it effectively - For flow run logs (visible in the UI on the flow run page): set the env var in the flow’s runtime environment (e.g., in your deployment/job template). - For worker logs: set the env var where you start the worker and read the worker’s stdout/stderr (host terminal, container logs, kubectl logs, CloudWatch, etc.). Examples - Start a worker with debug logs:

PREFECT_LOGGING_LEVEL=DEBUG prefect worker start -p <your-pool>

- Make your flow runs log at debug (configure env on the deployment/job template), then confirm in the flow run’s Logs tab. Related docs - Workers: Workers concept - Logging customization: Logging customization If you share what infrastructure your work pool uses (Process, Docker, Kubernetes, ECS, etc.), I can give exact steps to view the worker logs there.

Short answer: those timestamps are when the worker emitted those log lines. In the ECS worker, the sequence typically is: - “Creating ECS task run…” → the worker is preparing the AWS ECS RunTask call - “Task run request { … }” → the worker logs the request payload it’s about to send to ECS - A subsequent line with a taskArn (e.g., “Submitted ECS task … taskArn=…”) → AWS has responded to RunTask So the timestamp on “Task run request { … }” is effectively “just before the worker calls ECS RunTask”. The timestamp on the line that includes the taskArn is when the RunTask API returned. Tips to confirm and correlate: - Look for the next log line that includes

taskArn

— that marks the time AWS responded. - Cross-check with ECS events/CloudWatch or CloudTrail for “RunTask” to see the server-side time. - Remember these are worker logs; they won’t appear in the flow-run log stream. Related docs: - Workers concept: Workers concept - ECS guide: Prefect AWS ECS guide If you share a snippet after the “Task run request { … }” line, I can help point out the exact “AWS responded” log in your sequence.