Madushan Pathirana
04/07/2026, 11:58 AMMarvin
04/07/2026, 11:58 AMMarvin
04/07/2026, 12:05 PMprefect work-pool create "local-process" --type process
prefect worker start -p local-process
Then point your deployment at local-process.
- Very fast: Docker work pool (local Docker on the same EC2 box)
- Typical start time: ~0.5–2s with the image cached
- Tradeoff: needs Docker daemon on that EC2 host
- Steps:
uv run --with 'prefect[docker]' prefect work-pool create "local-docker" --type docker
uv run --with 'prefect[docker]' prefect worker start -p local-docker
In your deployment, use a pinned image tag and pre‑pull it on the host for immediate starts.
If you want to keep ECS (EC2 launch type) but cut latency as much as possible
- Reduce worker polling interval (default is 10s):
export PREFECT_WORKER_QUERY_SECONDS=2
- Ensure EC2 capacity is warm (ASG min capacity > 0, no scale‑up delay).
- Reuse task definitions to avoid registrations:
- In the ECS work pool variables, enable match_latest_revision_in_family: true or supply a fixed task_definition_arn.
- Keep images cached on instances:
- Use pinned tags (not latest) and pre‑pull on your EC2 instances; avoid “always pull” behavior.
What I recommend for your setup
- Since you run Prefect Server and workers on a single EC2 instance and all envs share that host, use:
- Process pool for the absolute fastest dev/test/startup.
- Docker pool for fast starts plus container isolation.
- Keep your ECS EC2 pool for production later if you need horizontal scale across instances.
Quick deployment example (process)
from prefect import flow
@flow
def my_flow():
...
if __name__ == "__main__":
my_flow.deploy(
name="ultra-fast",
work_pool_name="local-process",
)
Question: Is running flows directly on that same EC2 host acceptable from an isolation/compliance standpoint, or do you need them to run inside ECS tasks? That will determine whether we move you to Process/Docker or just aggressively tune your ECS pool.