Short answer: with ECS push pools, “Late” usually means Prefect tried to submit your run around its scheduled time but either had to wait/retry before AWS would accept it, or the server’s submit loop didn’t pick it up immediately. Even with no Prefect concurrency limits, AWS-side throttling/capacity and Prefect’s protective backoff can stretch that delay into several minutes. What commonly causes 5–10+ minute Late periods on ECS push pools - AWS throttling or capacity hiccups - RunTask/RegisterTaskDefinition can be throttled (TooManyRequests/Throttling) or blocked by capacity provider issues (Fargate spot unavailable, IP exhaustion in subnets, ENI limits, tasks-per-cluster quotas). - Prefect will retry with backoff on these errors; repeated provider rejections can add minutes before a successful submission. - Push pool submission cadence and backoff - Push pools are submitted by the Prefect server. If a burst of runs hit at once or recent provider errors occurred, the server will pace and retry submissions to avoid repeated throttling. - No Prefect concurrency limits ≠ no limits at all - AWS quotas (Fargate vCPU, tasks per cluster, ENIs/IPs in subnets, SG limits) still act like “hidden” concurrency caps and can force retries. - Normal “Late” window - Prefect marks runs Late shortly after their scheduled time if they haven’t started yet. Even in healthy conditions, there’s a small buffer from scheduling → Late → submission; with any of the above factors, that buffer can expand. How to confirm quickly - Check the run timeline/logs: before “Infrastructure started”, do you see repeated “Submitting to ECS” followed by retry messages or delays? - Inspect the ECS cluster in CloudWatch/ECS Events around the same timestamps for: - Throttling exceptions - “Unable to place a task” / “no capacity” messages - IP/subnet exhaustion or ENI limits - Look for other effective limits: - Tag-based or global concurrency limits in Prefect (even if pool/queue limits are None) - AWS quotas: Fargate vCPU, tasks-per-service/cluster, ENIs per subnet/region Mitigations that usually help - Ensure AWS capacity: - Increase Fargate vCPU/ENI quotas; use multiple subnets with sufficient IPs; avoid spot for critical runs or add on-demand capacity. - Reduce ECS API pressure: - Reuse a stable task definition instead of creating a new one per run (cuts down on RegisterTaskDefinition calls). - Smooth burst scheduling (stagger start times) so runs don’t all submit at once. - Prefer pull-based workers if you need lower-latency fan-out - A pull (worker)–based ECS pool can parallelize submissions on your infra and isolate backoff to individual workers. - Turn up logging while you diagnose - If you’re self-hosting, set server logs to debug and watch for ECS submission retries/backoff. - In Cloud, check the Work Pool/Run logs around Late periods and correlate with ECS events. If you can share a recent example run ID, your Prefect environment (Cloud vs self-hosted), and ECS details (Fargate vs EC2, region, typical burst size), I can help pinpoint which of the above is biting you and suggest exact changes.

Yes — you can confirm this pretty directly in AWS. Best places to look in the AWS console: 1. CloudTrail Event History This is the most useful place to prove API throttling. - Go to CloudTrail → Event history - Filter by: - Event source:

<http://ecs.amazonaws.com|ecs.amazonaws.com>

/

<http://elasticcontainerservice.amazonaws.com|elasticcontainerservice.amazonaws.com>

- Event name:

RunTask

,

DescribeTasks

, and sometimes

RegisterTaskDefinition

- Look for failed events with errors like: -

ThrottlingException

-

Rate exceeded

-

TooManyRequestsException

-

RequestLimitExceeded

If those show up at the same timestamps as the delay before Infrastructure started, that’s strong evidence of AWS API throttling. 2. ECS cluster/task events In ECS → Clusters → <your cluster>: - Check the Tasks tab for the task Prefect launched - Open the task and inspect: - state transitions - stopped reason - task events / service events if applicable If Prefect says it submitted the run, but ECS shows the task appearing late or moving slowly through

PROVISIONING

/

PENDING

, that helps narrow it down to AWS-side delay rather than Prefect orchestration. 3. CloudWatch Logs Check logs for both: - the Prefect worker that is calling ECS - the ECS task once it starts In the worker logs, search for: -

Submitting to ECS

- retry messages -

throttl

-

Rate exceeded

If the worker is retrying AWS API calls, you’ll usually see that here before the task ever starts. 4. Service Quotas Go to Service Quotas and inspect ECS/Fargate-related quotas in the affected region. Useful ones to review: - ECS API request quotas - Fargate vCPU/resource quotas - task count/concurrency-related limits If you’re launching many flow runs at once, sometimes the issue is not

RunTask

API throttling specifically, but account-level Fargate capacity quotas. 5. CloudWatch metrics / usage spikes In CloudWatch, check whether the timing of the delays lines up with: - bursty task launches - spikes in worker activity - ECS API errors if you have them surfaced in logs/metrics If you have many runs submitting at the same time, that pattern is a common cause. What to correlate specifically Try lining up these timestamps: - Prefect log line:

Submitting to ECS

- any retry/backoff log lines - CloudTrail

RunTask

/

DescribeTasks

events - ECS task creation timestamp - Prefect

Infrastructure started

If there’s a large gap between Prefect submission and the first successful ECS task event, AWS-side throttling or quota pressure is very likely. If you want, I can also give you a step-by-step checklist for CloudTrail filters or help interpret a specific worker log snippet.

Yes — for a work pool concurrency limit, slots should be freed as soon as those runs leave active states. There is not a decay/cooldown mechanism on work pool concurrency. So in your example: - work pool concurrency limit =

50

- you submit

200

runs Behavior should be: - up to

50

runs can be active at once - the other

150

wait in the queue - when one of the first

50

finishes, that slot becomes available - another queued run can be picked up immediately So if removing the work pool limit fixed the “late task” behavior, that strongly suggests the limit itself was the bottleneck. A couple of nuances: - Work pool / work queue concurrency is different from global concurrency limits, which can have decay-like behavior depending on configuration. - With work pool concurrency, the usual reason for delay is simply that queued runs are waiting for an open slot, not that Prefect is intentionally holding slots after completion. What probably happened in your experiment If you launched a large burst, Prefect likely kept only

50

runs eligible to start, and the rest stayed queued. If those queued runs crossed your “late” threshold before getting picked up, they would appear late even though the system was behaving as configured. What to check In the UI/API, confirm whether the delayed runs were staying in a queued/scheduled state while the first batch occupied the pool slots. If yes, that points squarely at the pool limit. Docs: - Work pools - Global concurrency limits If you want, I can also help you think through what concurrency limit is safe for ECS so you avoid both Prefect queueing and AWS throttling.

Yes — that can happen, and the key point is: free work pool slots do not themselves “push” late runs into execution immediately. A flow run will stay

Late

until a worker actually polls, claims, and submits it. So even if the pool is now empty, there can still be a lag before those late runs get picked up. Most likely causes: 1. Worker polling cadence Workers poll the API on an interval; they do not get instantly notified when slots free up. If the worker is under load, backing off, or polling less frequently than expected, runs can sit in

Late

even though capacity is available. 2. Worker-side submission backlog Even after a worker sees available runs, it still has to submit them to ECS. If ECS submissions are slow, batched, retried, or otherwise delayed, the runs can remain late for a while before transitioning forward. 3. Queue ordering / prefetch behavior Workers ask for runnable runs in batches. If there was a large burst, some runs may not be claimed on the first poll even after slots free up. 4. State transition timing A run is marked

Late

when its scheduled start time has passed and it still hasn’t been picked up. It won’t stop being late just because the pool now has room — it stops being late once the worker advances it. So your observation is consistent with: - the work pool limit causing runs to queue - then, after slots freed up, workers taking additional time to come back around and pull those runs That’s why removing the concurrency limit can appear to “fix” it: you eliminated the queueing point, so fewer runs ever sit around long enough to become late. What I’d check next - Worker logs around the delayed period: - are workers polling regularly? - do you see long gaps between polling/submission cycles? - do you see bursts of

Submitting to ECS

only every few minutes? - Number of workers for that pool: - if only one worker is serving a busy ECS pool, it may just be draining queued work too slowly - Worker settings: -

PREFECT_WORKER_QUERY_SECONDS

-

PREFECT_WORKER_PREFETCH_SECONDS

- Whether ECS submission is slow even after the worker picks up the run Docs: - Workers - Work pools If you want, I can help you reason through a concrete example like: - 1 worker - pool concurrency 50 - 200 submitted runs - average runtime X minutes and estimate whether the 10–15 minute late window matches worker polling/drain behavior or suggests something else.

Yes — slightly, but not in the way that changes the core answer. For an ECS push work pool, the important difference is: - Prefect is pushing runs directly to ECS - there is not a separate long-lived worker process you manage that polls like a classic worker But the same practical idea still applies: freeing work pool concurrency does not instantly launch all late runs. There is still an orchestration loop on the Prefect side that has to: 1. notice eligible runs 2. apply concurrency rules 3. submit them to ECS 4. wait for AWS-side submission/startup So with ECS push pools, the delay is less about your worker’s local polling loop and more about: - Prefect’s orchestration/scheduling cycle - queue draining behavior - AWS submission latency - ECS/API throttling or backpressure So yes, it is slightly different operationally, but the symptom can look the same: - runs sit

Late

- slots are technically free - runs are not submitted immediately For ECS push pools specifically, I’d suspect these contributors: 1. Prefect-side scheduling/orchestration cadence Even with slots free, the backend still has to move scheduled/late runs into submission. 2. Batch draining behavior If a large number of runs became eligible at once, Prefect may not drain them all instantly. 3. ECS submission latency Even after Prefect decides to submit, ECS

RunTask

and related AWS calls can add delay. 4. AWS throttling/backoff This is especially relevant if removing the concurrency limit changed the submission pattern. One important nuance: if all pool slots are free again and runs still remain

Late

for 10–15 minutes, that suggests the bottleneck is not just slot release itself. At that point I’d look at: - delayed orchestration in Prefect Cloud/server - ECS push submission backlog - AWS throttling / retries - whether those runs are still in `Scheduled`/`Late` vs already moved to

Pending

So I’d frame it like this: - Classic worker pool: delay can often be explained by worker polling/submission cadence - ECS push pool: delay is more likely in Prefect’s push submission path and/or AWS ECS acceptance/startup path If helpful, I can help you build a precise state-based diagnosis for ECS push pools, e.g.: - if run is

Late

+ no ECS task exists yet → likely Prefect-side submission backlog - if run is no longer `Late`/is

Pending

but ECS task starts much later → likely AWS/ECS-side delay - if many runs transition in bursts every few minutes → likely orchestration batching or throttling