Alastair
04/27/2026, 3:54 PMMarvin
04/27/2026, 3:54 PMMarvin
04/27/2026, 3:56 PMecs:push pool for jobs where startup latency is acceptable
- Add a second work pool for lower-latency jobs
- Run a long-lived worker against that second pool
- Point specific deployments at the pool that fits their needs
So yes, it can be as simple as “create another work pool and run a worker that polls it”.
A couple of important clarifications though:
### 1. If you want faster than ECS task startup, an ECS worker may not help much
If the worker is polling a normal ecs work pool, it still launches ECS tasks for flow runs. That removes the push mechanism, but not the underlying ECS startup latency.
So there are two different patterns:
#### Option A: ecs:push + ecs worker
- Both ultimately run flow runs on ECS
- Useful if you want more control over infra / networking / credentials / execution model
- Does not fundamentally eliminate ECS startup time
#### Option B: ecs:push + process / docker / kubernetes worker on always-on infra
- Worker runs continuously on a VM, ECS service, K8s pod, etc.
- Flow runs start as subprocesses or containers on already-running infrastructure
- This is the pattern that usually helps with low-latency execution
Based on your goal, Option B sounds more like what you want.
### Recommended architecture
For your case, I’d usually suggest:
- Pool 1: existing ecs:push pool
- for heavier / bursty / less latency-sensitive jobs
- Pool 2: a process or docker work pool
- backed by a long-running worker on always-on compute
- for quick jobs you want to start fast
Then each deployment chooses its pool via work_pool_name.
### Good choices for the “fast” side
#### process pool
Best if:
- the worker machine already has your code + dependencies
- you want the fastest startup
- you’re okay running flows as subprocesses on that machine
#### docker pool
Best if:
- you want container isolation
- you still want fairly fast startup
- you have a host running Docker continuously
#### kubernetes pool
Best if:
- you already have K8s
- pod startup is acceptable
- you want stronger orchestration/isolation
### What setup looks like conceptually
1. Create a new work pool
2. Start a worker polling that pool
3. Create deployments that target that pool
For example, your quick jobs would use one pool, and your existing ECS-backed jobs would continue using the other.
Example deployment idea:
python
from prefect import flow
@flow
def quick_task():
print("run quickly")
@flow
def heavy_task():
print("run on ECS push")
if __name__ == "__main__":
quick_task.deploy(
name="quick-task",
work_pool_name="fast-process-pool",
)
heavy_task.deploy(
name="heavy-task",
work_pool_name="ecs-push-pool",
image="your-image:latest",
)
### Operational caveat
If you use a process worker, that worker host becomes part of your runtime:
- it needs your code available
- it needs Python deps installed
- you’ll want to think about concurrency limits and resource contention
- HA usually means running more than one worker / more than one host
### Tooling
Prefect supports this natively with work pools + workers; you don’t need special orchestration beyond that.
Useful docs:
- Work pools
- Workers
- Deployments
- Manage work pools
- Deploy via Python
- Serverless / push pools
- ECS worker guide
### Practical recommendationMarvin
04/27/2026, 3:56 PMprocess work pool on a small always-on instance, or
- docker work pool on an always-on Docker host
That usually gives the “quick lane” you’re looking for.
### Bottom line
Yes, a hybrid setup is a normal Prefect pattern.
But the key distinction is:
- new work pool + worker is the right mechanism
- using ECS again probably won’t solve the latency issue
- for true fast-start jobs, prefer a long-running process or docker worker on warm infrastructure
If you want, I can sketch a concrete setup for:
- ECS push + process worker, or
- ECS push + docker worker,
including how to structure the deployments cleanly so you don’t duplicate config.Alastair
04/27/2026, 4:00 PMMarvin
04/27/2026, 4:00 PMMarvin
04/27/2026, 4:02 PMprefect-aws, but that tooling is for deploying an ECS worker that polls an ecs work pool — i.e. a worker whose job is to launch other ECS tasks for flow runs.
That is not the same as:
- running a generic Prefect worker on ECS
- polling a process work pool
- executing flow runs as local subprocesses inside that long-running ECS task
For your design, Prefect’s role is basically just:
1. create a process work pool
2. run a worker with:
bash
prefect worker start --pool <your-process-pool>
and you choose where that worker process lives.
### So what tooling exists?
#### Built-in Prefect tooling
- Yes for: deploying/managing an ECS worker for an ecs work pool via prefect-aws
- No for: “take this generic worker and create an ECS service for it on my existing cluster”
So if you want a long-running ECS task that runs a process worker, you’d generally provision that ECS service/task definition yourself using your normal AWS/IaC tooling:
- Terraform
- CloudFormation / CDK
- Copilot
- console/manual ECS service setup
### What your ECS task would do
Your container entrypoint/command is just the worker start command, something like:
bash
prefect worker start --pool fast-process-pool
with environment variables such as:
- PREFECT_API_URL
- PREFECT_API_KEY
and of course your image needs:
- your flow code
- your Python dependencies
- Prefect installed
### Important caveat for process on ECS
This setup works, but remember what process means:
- flow runs execute as subprocesses inside the same ECS task/container that is running the worker
- so the worker container is also the execution environment
- concurrency means multiple subprocesses on that same task, subject to CPU/memory limits
That’s why this can be great for low-latency jobs, but you’ll want to size that worker task appropriately and probably set a worker concurrency limit.
### When this pattern makes sense
It’s a good fit if you want:
- warm always-on runtime
- minimal startup overhead
- code/image already present
- “quick lane” execution
It’s less ideal if you want:
- strong per-run isolation
- per-run autoscaling
- very different resource shapes per job
### If you want the cleanest setup
Your architecture would look like:
- ecs-push-pool
- for normal/heavier jobs
- fast-process-pool
- polled by a long-running ECS service running prefect worker start --pool fast-process-pool
That’s a very reasonable hybrid design.
### Relevant docs
- Work pools
- Workers
- Deployments
- ECS worker integration
### Bottom line
No, Prefect does not currently provide a dedicated “deploy a generic/process worker as an ECS service on my cluster” tool.
For your case, the usual path is:
- create the process work pool in Prefect
- create an ECS service yourself using your existing infra tooling
- have that service run prefect worker start --pool ...
If you want, I can sketch:
- the minimal ECS task definition/service shape for this, or
- the tradeoffs between running one process worker task with concurrency vs multiple worker tasks with low concurrency each.