https://prefect.io
Join Slack
<@ULVA73B9P> our prefect ecs worker gets overwhelmed when we have hundreds of flow runs waiting on ...
# ask-marvin
j
@Marvin our prefect ecs worker gets overwhelmed when we have hundreds of flow runs waiting on deployments with concurrency limits. then other flow runs in deployments that dont have limits get stuck in a Late state instead of just running. how do we fix this?
m
✅ thought for 179.7 seconds
Yes — this is usually a work-pool/worker starvation problem, not just a deployment-concurrency problem. What’s happening is roughly: - your ECS worker polls a pool/queue for available flow runs - a large backlog from deployments with concurrency limits dominates what the worker sees - the worker spends its available submission capacity on those runs / that queue - runs from unconstrained deployments don’t get picked up quickly enough, so they sit in 
Late
Best fix: isolate those workloads so one noisy constrained deployment can’t monopolize the same worker capacity. Recommended approaches, in order: 1. Split deployments into separate work pools or at least separate work queues - Put constrained/high-backlog deployments in one pool/queue - Put unconstrained/latency-sensitive deployments in another - Run separate workers for each This is the cleanest fix for ECS. 2. Set concurrency at the work-pool or work-queue level too - Deployment concurrency limits alone don’t provide fairness across unrelated deployments - Queue/pool limits help prevent one class of work from consuming all worker attention 3. Lower worker submission pressure - Reduce worker 
--limit
if the ECS worker is spawning too many tasks at once - Reduce prefetch seconds so the worker doesn’t grab too much work too early 4. Use queue priority intentionally - Put the deployments that should start quickly in a higher-priority queue - Put backlog-heavy deployments in a lower-priority queue For example, a good ECS layout is: - 
ecs-batch-pool
for deployments with deployment concurrency limits / long backlogs - 
ecs-realtime-pool
for unconstrained or low-latency deployments Then run separate workers:
Copy code
bash
prefect worker start --pool ecs-batch-pool --limit 10
prefect worker start --pool ecs-realtime-pool --limit 20
If you want to keep one pool, use separate work queues instead: - 
realtime
queue: higher priority - 
batch
queue: lower priority, optionally with its own queue concurrency limit Why this helps: - ECS task startup is relatively slow - a single worker can get tied up repeatedly submitting backlog-heavy runs - once its effective capacity is occupied, other runs remain 
Scheduled
long enough to become 
Late
So the practical answer is: don’t mix high-backlog constrained deployments with unconstrained deployments on the same worker path if you care about fairness/latency. A few extra knobs to consider: - decrease worker prefetch via 
PREFECT_WORKER_PREFETCH_SECONDS
- tune poll frequency via 
PREFECT_WORKER_QUERY_SECONDS
- keep worker 
--limit
modest for ECS, since task launches are not instantaneous If you want, I can give you a concrete recommended topology for your setup, e.g.: - single ECS cluster + multiple Prefect pools/queues, or - exact queue priority/concurrency settings based on how many runs you expect.
j
@Marvin sure. let's start with the assumption we're constrained to a single work pool. we could potentially add more workers or scale up the compute a single worker
m
✅ thought for 9.2 seconds
Got it — if you’re staying on a single work pool, then the best fix is to add work-queue isolation + worker isolation within that pool. What I’d recommend 1. Create at least two work queues in the same pool - 
realtime
or `default`: for deployments that should start promptly - 
backlog
or `limited`: for deployments with deployment concurrency limits and large scheduled backlogs 2. Give the realtime queue higher priority - Higher priority queues are checked first - Put unconstrained / latency-sensitive deployments there 3. Run separate workers against different queues in the same pool - Worker A polls only 
realtime
- Worker B polls only 
backlog
That gives you fairness without needing a second pool. Conceptually:
Copy code
text
single work pool
├── realtime queue   (high priority)
└── backlog queue    (lower priority)

worker-1 -> realtime queue only
worker-2 -> backlog queue only
Why this works Right now, one ECS worker is likely spending most of its submission capacity on backlog-heavy deployments. Even if other deployments have no concurrency limit, they still compete for the same worker attention. Splitting queues and workers prevents that contention. Suggested tuning For ECS specifically, I’d avoid one giant worker with a very high limit. Instead of: - 1 worker with 
--limit 100
prefer something more like: - 1 worker for 
realtime
with 
--limit 10-20
- 1 worker for 
backlog
with 
--limit 5-10
Why: - ECS task launches are slow enough that very large per-worker limits often increase contention rather than throughput - smaller worker limits make behavior more predictable - dedicated capacity guarantees unconstrained deployments always have room If you can add workers, do that before scaling a single worker way up Multiple smaller workers is usually better than one huge worker for fairness and failure isolation. Deployment mapping Put deployments like this: - unconstrained / user-facing / frequent schedules -> 
realtime
- deployments with 
concurrency_limit
and hundreds of waiting runs -> 
backlog
Important note If both kinds of work remain on the same queue, adding compute or increasing a single worker’s limit may help throughput a bit, but it usually does not fix fairness. The noisy deployments can still dominate pickup. So within your single-pool constraint, the highest-value change is: single pool, multiple queues, dedicated workers per queue If you want, I can next sketch: - the exact CLI commands to create the queues and start workers, or - a 
flow.deploy(...)
example showing how to assign deployments to each queue.
j
@Marvin how do i define all of this in my prefect.yaml
m
✅ thought for 133.3 seconds
Yes — you can define which deployment goes to which queue in 
prefect.yaml
. The main thing to know is: - work pool assignment goes in 
work_pool.name
- work queue assignment goes in 
work_pool.work_queue_name
- queue properties like priority/concurrency are not defined in `prefect.yaml`; set those separately in the UI/CLI/API Example for your case:
Copy code
yaml
name: my-project
prefect-version: 3.0.0

deployments:
  - name: unconstrained-fast
    entrypoint: flows/fast.py:fast_flow
    work_pool:
      name: my-ecs-pool
      work_queue_name: realtime
      job_variables:
        image: my-image:latest

  - name: constrained-backlog
    entrypoint: flows/batch.py:batch_flow
    concurrency_limit: 5
    work_pool:
      name: my-ecs-pool
      work_queue_name: backlog
      job_variables:
        image: my-image:latest
That gives you: - 
unconstrained-fast
-> 
my-ecs-pool
/ 
realtime
- 
constrained-backlog
-> 
my-ecs-pool
/ 
backlog
Then run separate workers targeting those queues. A few important caveats: 1. The work pool must already exist 
prefect.yaml
does not create the work pool for you. 2. The work queue name can be referenced in 
prefect.yaml
 If the queue does not already exist, Prefect can create it when the deployment is created. 3. But if you want queue priority or queue concurrency limits, create/configure the queues explicitly That part is not controlled in 
prefect.yaml
. So your likely workflow is: 1. create the ECS work pool 2. create/configure the two queues 3. keep deployment-to-queue mapping in 
prefect.yaml
4. run dedicated workers per queue A practical layout would be:
Copy code
yaml
name: my-project
prefect-version: 3.0.0

deployments:
  - name: api-ish-work
    entrypoint: flows/api.py:api_flow
    work_pool:
      name: my-ecs-pool
      work_queue_name: realtime
      job_variables:
        image: my-image:latest

  - name: scheduled-batch
    entrypoint: flows/batch.py:batch_flow
    concurrency_limit: 10
    work_pool:
      name: my-ecs-pool
      work_queue_name: backlog
      job_variables:
        image: my-image:latest
Then configure queue behavior outside the file: - `realtime`: higher priority - `backlog`: lower priority, possibly a queue concurrency limit too One more note: if you omit 
work_queue_name
, the deployment goes to the pool’s 
default
queue. If you want, I can next give you: - a full 
prefect.yaml
example with build/push/pull steps for ECS, or - the exact worker commands you’d use for 
realtime
and 
backlog
.
j
@Marvin how do i target the worker at the work queue?
m
✅ thought for 12.8 seconds
Use 
prefect worker start
with 
--work-queue
. Example:
Copy code
bash
prefect worker start --pool my-ecs-pool --work-queue realtime
And for the backlog queue:
Copy code
bash
prefect worker start --pool my-ecs-pool --work-queue backlog
So if you want dedicated workers:
Copy code
bash
prefect worker start --pool my-ecs-pool --work-queue realtime --limit 15
prefect worker start --pool my-ecs-pool --work-queue backlog --limit 5
A few notes: - 
--pool
is required - 
--work-queue
restricts that worker to one or more queues in the pool - if you omit 
--work-queue
, the worker will poll all queues in the pool You can also target multiple queues with repeated flags:
Copy code
bash
prefect worker start --pool my-ecs-pool --work-queue realtime --work-queue default
For your case, though, I’d keep them separate so the backlog queue can’t interfere with the realtime queue. If you want, I can also show how to do this in: - ECS task definitions / container args - Helm / Kubernetes - systemd / Docker Compose
j
@Marvin explain again what the limit means
m
✅ thought for 8.9 seconds
--limit
is the maximum number of flow runs that a single worker will have in progress at once. For example:
Copy code
bash
prefect worker start --pool my-ecs-pool --work-queue realtime --limit 15
means: - that worker will submit/pick up at most 
15
flow runs at a time - once it has 15 in progress, it won’t take more until some finish or otherwise release capacity For an ECS worker, “in progress” effectively means runs that this worker has already handed off to ECS and is still tracking. So 
--limit
is worker-side concurrency, not deployment-side concurrency. These are different: - deployment 
concurrency_limit
 = max concurrent runs for one deployment - worker 
--limit
 = max concurrent runs handled by one worker process - work queue/work pool concurrency limits = broader limits across that queue/pool Why it matters for you: If you have one worker with:
Copy code
bash
--limit 100
then that worker can spend a lot of its capacity servicing one noisy queue/deployment backlog. If instead you run:
Copy code
bash
prefect worker start --pool my-ecs-pool --work-queue realtime --limit 15
prefect worker start --pool my-ecs-pool --work-queue backlog --limit 5
then you’ve effectively reserved: - up to 15 concurrent flow runs for 
realtime
- up to 5 concurrent flow runs for 
backlog
That’s why this helps fairness. One subtle point: 
--limit
does not mean “CPU/memory limit” for the worker container itself. It’s not a compute resource setting. It’s just how many flow runs that worker will manage concurrently. If you want, I can also explain how 
--limit
interacts with: - ECS cluster capacity - deployment concurrency limits - queue priority because those three together are the full picture.
j
@Marvin we fan out pretty aggressively this seems not ideal for us?
m
✅ thought for 8.5 seconds
Yeah — if you intentionally fan out very hard, small worker 
--limit
values can become a bottleneck. So the tradeoff is: - higher 
--limit
 => better throughput / faster burst pickup - lower 
--limit
 => better fairness / less chance one class of work starves another For your setup, I would not treat 
--limit
as the primary fix. If high fan-out is important, the better fix is still: single pool, multiple queues, multiple workers, with queue isolation That way you can keep total throughput high without letting backlog-heavy deployments starve everything else. Example:
Copy code
bash
prefect worker start --pool my-ecs-pool --work-queue realtime --limit 50
prefect worker start --pool my-ecs-pool --work-queue backlog --limit 100
That gives you: - large fan-out capacity - dedicated capacity for both categories - no competition between realtime and backlog queues at the worker level So instead of: - one worker at 
--limit 150
prefer: - one worker for 
realtime
at 
--limit 50
- one worker for 
backlog
at 
--limit 100
or even multiple workers per queue. This is usually better because: - you preserve burst capacity - you improve fairness - you avoid a single worker becoming the choke point - ECS task launch overhead is spread across workers Important nuance: if your ECS cluster itself is the real bottleneck, then increasing worker limits or worker count won’t help much. In that case: - Prefect may schedule lots of runs - ECS may still delay actual task placement - runs can still appear late for infrastructure reasons So there are really two separate capacities: 1. Prefect pickup/submission capacity - worker count - worker 
--limit
- queue isolation 2. ECS execution capacity - available cluster/Fargate capacity - task placement speed - AWS API throttling / startup latency If your complaint is specifically: - “unconstrained deployments get stuck Late while constrained deployments flood the system” that strongly points to Prefect-side pickup fairness, which queue-specific workers solve better than just scaling one worker up. So my refined recommendation for an aggressive fan-out system is: - keep one work pool if needed - split into at least 2 queues - run dedicated workers per queue - keep limits reasonably high, not tiny - scale horizontally with more workers before making one worker enormous If you want, I can help you choose actual numbers for: - number of workers - per-worker 
--limit
- queue split based on roughly how many flow runs you burst per minute and how long ECS tasks take to start.
j
@Marvin if we scope a worker to a single queeue, which is scoped to a single deployment, does syncing its limit w the concurrency limit of that deployment make sense? what happens to the jobs that are scheduled not witin the limit?
m
✅ thought for 10.0 seconds
Yes — if a worker is dedicated to one queue, and that queue is effectively dedicated to one deployment, then setting the worker 
--limit
roughly equal to that deployment’s 
concurrency_limit
can make sense. That alignment means: - deployment limit says: “only 
N
runs of this deployment may run at once” - worker limit says: “this worker will only try to manage about 
N
runs at once” So you avoid the worker trying to over-pull work it can’t actually progress. But whether it’s best depends on what you want: - if the queue truly only serves that one deployment: yes, often sensible - if the queue may later serve multiple deployments: less ideal, because the worker limit becomes a hidden shared bottleneck What happens to runs scheduled beyond the deployment concurrency limit? Those runs do not run immediately. With deployment concurrency configured to enqueue excess runs, the extra runs wait until a slot opens. In practice, they remain queued by Prefect rather than being launched onto ECS right away. As capacity becomes available, additional scheduled runs are allowed through. So if: - deployment concurrency limit = 
10
- worker limit = 
10
- 200 runs are scheduled then roughly: - up to 10 can progress concurrently - the rest wait for capacity - as one finishes, another waiting run can move forward Those extra runs are not lost. They remain pending/scheduled/queued in Prefect until they can acquire concurrency. Why syncing the two limits can help If you set:
Copy code
bash
prefect worker start --pool my-ecs-pool --work-queue backlog --limit 10
and the deployment also has:
Copy code
yaml
concurrency_limit: 10
then the worker won’t try to fan out far beyond the actual allowed concurrency for that deployment. That usually reduces churn and makes behavior more predictable. But there’s an important caveat If your deployment concurrency limit is implemented by Prefect as “only let 10 active at a time, keep the rest waiting”, then a worker limit equal to deployment concurrency is fine. If you instead set worker limit lower than deployment concurrency, like: - deployment concurrency = 50 - worker 
--limit
= 10 then the worker becomes the bottleneck, and you’ll only get 10 active runs even though the deployment would allow 50. So the rules of thumb are: - worker limit < deployment concurrency -> worker throttles more than deployment does - worker limit ~= deployment concurrency -> good alignment for a single-deployment queue - worker limit > deployment concurrency -> deployment limit remains the real cap What state are the extra runs in? The exact state can vary based on Prefect’s concurrency handling, but conceptually they are waiting for a deployment concurrency slot rather than being executed. They are not launched to ECS until Prefect allows them through. So to your specific question: Yes, aligning the worker limit with the deployment concurrency limit is a reasonable pattern when that worker/queue is dedicated to that one deployment. If you want, I can draw a concrete timeline for: - 1 deployment - concurrency limit 5 - worker limit 5 - 20 scheduled runs showing exactly how the runs move through states.
j
@Marvin how much compute do i need for a given worker
m
✅ thought for 10.6 seconds
There isn’t a clean fixed CPU/RAM formula for a Prefect worker. For an ECS worker, the worker mostly does control-plane work: - polls Prefect for runnable flow runs - renders job configuration - calls AWS ECS APIs to submit tasks - tracks submitted runs / heartbeats So the worker usually does not need compute proportional to the flow runtime itself. The flow-run ECS tasks need the real compute; the worker usually needs enough resources to keep up with orchestration. Rule of thumb: the worker is usually light on CPU/memory, until you make it handle very high submission concurrency. What drives worker sizing most: 1. polling frequency 2. number of runs being submitted concurrently 3. how bursty your schedules are 4. how expensive your job templating/env rendering is 5. AWS API latency/retries For most ECS workers, I’d start pretty small and scale from observations: - small/medium workload: 
0.25-0.5 vCPU
, 
512MiB-1GiB RAM
- higher burst / aggressive fan-out: 
0.5-1 vCPU
, 
1-2GiB RAM
Usually memory is not the issue first; API throughput / event loop responsiveness is. What to watch for if the worker is underprovisioned: - worker logs show slow polling or delayed submission - lots of runs become 
Late
even though ECS has capacity - high CPU on the worker container - worker restarts / OOMs - slow task submission during burst windows A practical sizing strategy: Start with something like: - 
0.5 vCPU
- 
1 GiB RAM
Then increase if you see the worker struggle during bursts. Scale workers horizontally before making one worker huge If you need more throughput, usually better to run: - 2 workers at modest size than: - 1 oversized worker because that also improves queue isolation and failure tolerance. Important distinction If your flow runs themselves need lots of compute, that does not mean the worker does. The worker just launches ECS tasks; the actual ECS task definitions for the deployments should carry the heavier CPU/RAM requirements. So if you ask “how much compute do I need for a given worker limit?”, the honest answer is: - there is no strict 1:1 mapping like “10 runs = 1 vCPU” - worker sizing depends more on launch rate than run count For example: - 
--limit 100
but starts spread over an hour -> worker may be fine - 
--limit 100
with 100 runs due in the same minute -> worker needs more headroom A useful mental model: - worker compute scales with how fast it must submit runs - ECS task compute scales with how heavy the flows are If you want, I can help you build a rough sizing table like: - X runs/minute burst - Y ECS task startup latency - Z workers / per-worker CPU / 
--limit
for your workload.
j
@Chad Becker @Alexander Mezga @Colton Flowers @Anoop Sypereddi can you skim this please so we're all up to speed on the worker setup for prefect?
im going to set up queues and scoped workers, like suggested by marvin
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by