Short answer: seeing “Late” on everything beyond your pool’s concurrency is expected; those runs should still be submitted as slots free up. It is not expected that they never run. What’s going on - In Prefect 3.x, a work pool concurrency limit (e.g., 10) caps how many flow runs in that pool can be PENDING/RUNNING at once. - If you trigger 100 runs at the same time, the first 10 will move to PENDING/RUNNING. The remaining 90 will sit in SCHEDULED and quickly flip to “Scheduled (Late)” because their scheduled time has passed. “Late” is informational and does not block submission. - As soon as any of the first 10 leave PENDING/RUNNING, the next SCHEDULED/Late runs should be submitted. If they never are, something else is holding capacity. Common reasons remaining runs “never run” - The initial 10 are stuck in PENDING (e.g., ECS capacity/permissions/job template issues), so the pool stays at its concurrency cap. - Additional concurrency constraints exist (e.g., deployment-level concurrency with a collision strategy). - The pool (or relevant work queue) is paused, or the push submitter cannot create new infra due to misconfig. Quick checks - Verify pool status and limits:

prefect work-pool inspect "<your-ecs-push-pool>"

- Check queues (and any queue-level limits):

prefect work-queue inspect -p "<your-ecs-push-pool>"

- Inspect a “stuck” flow run to see its last transition/reason:

prefect flow-run inspect <flow-run-id>

- Temporarily raise the pool limit to see if runs advance:

prefect work-pool set-concurrency-limit "<your-ecs-push-pool>" 20

Next steps - Do the first 10 runs complete, or are they lingering in PENDING? If they linger, check ECS task starts/logs and your pool’s base job template for issues. - Do you have deployment-level concurrency set on these deployments? - Are you on Prefect Cloud or self-hosted Server, and which Prefect version? If you can share: - Pool name/type, Prefect version, Cloud vs Server - Whether the initial 10 complete - An example flow run ID that stayed Late after earlier runs finished I can help pinpoint whether this is capacity saturation vs. a configuration issue.

Thanks for the details — that helps. This is not expected: in an ECS push work pool on Prefect Cloud 3.6.4, runs that go Scheduled (Late) should still be submitted as pool capacity frees up. Since your first 10 complete, the next runs should transition from Scheduled/Late → Pending → Running. Let’s quickly triage what could be blocking submission: Checks to run - Confirm the pool and queues aren’t paused and check their limits

prefect work-pool inspect "<your-ecs-push-pool>"
  prefect work-queue ls -p "<your-ecs-push-pool>" --verbose

If you see multiple queues, inspect the one your deployment targets:

prefect work-queue inspect -p "<your-ecs-push-pool>" "<queue-name>"

- Inspect the specific run’s state history to see if it ever tried to move to Pending

prefect flow-run inspect 069b7d8e-618d-7497-8000-38df03b828f6
  prefect flow-run logs 069b7d8e-618d-7497-8000-38df03b828f6

- Check the deployment for any deployment-level concurrency (collision strategy) that could block new submissions

prefect deployment inspect "<flow-name>/<deployment-name>"

What I’m looking for - Queue-level concurrency limits at or near 0 (these can block submissions even if the pool has room). - A paused queue (runs targeted to a paused queue will stay Scheduled/Late). - Deployment-level concurrency set to ENQUEUE or CANCEL_NEW that may be gating runs. - State history that never attempts Scheduled → Pending after earlier runs finished (would suggest a selection/submission issue). If everything above looks healthy, please share: - The work pool name, the queue name your deployment targets, and whether that queue is paused/limited - The deployment name for these runs - Your workspace slug (feel free to DM if you’d prefer) With that, I can escalate to our team using the run ID you provided and take a closer look on the backend.

Perfect — send it over and I’ll dig in. For the DM, please include: - Work pool and queue details - Output of:

prefect work-pool inspect "<your-ecs-push-pool>"
    prefect work-queue ls -p "<your-ecs-push-pool>" --verbose

If there’s a specific queue your deployment targets:

prefect work-queue inspect -p "<your-ecs-push-pool>" "<queue-name>"

- Deployment details (to check deployment-level concurrency/collision strategy)

prefect deployment inspect "<flow-name>/<deployment-name>"

- The run(s) that stayed Late after the first 10 completed

prefect flow-run inspect 069b7d8e-618d-7497-8000-38df03b828f6

- Rough timestamps: when the initial 10 completed, and when you noticed the next batch not progressing - Your Prefect Cloud org/workspace slug (feel free to DM privately) Once I have that, I’ll trace why capacity didn’t get re-allocated and get you a fix or a clear next step.

Got it — thanks! I’m reviewing your DM now and will dig into that specific run in your workspace. While I investigate, two quick diagnostics you can run on your side: - Trigger an artificial poll of the targeted work queue (helps confirm the selector sees your Scheduled/Late runs):

prefect work-queue read-runs -p "<ecs-push-pool-name>" "<queue-name>"

- If you want to sanity-check that slots are being reallocated, temporarily bump pool concurrency and see if one Late run advances:

prefect work-pool set-concurrency-limit "<ecs-push-pool-name>" 11

I’ll follow up shortly with what I find on: - Whether the deployment is targeting a paused/limited queue - Any queue-level concurrency that’s capping submissions - The state history for 069b7d8e-618d-7497-8000-38df03b828f6 and whether it attempted Scheduled → Pending after earlier runs finished Reference docs in case helpful: - Work pools and queues: Work pools (Prefect 3.x) - Concurrency limits: Concurrency limits - ECS push overview: ECS deployments (push) I’ll report back as soon as I’ve traced the run and queue state.