Fugo Takefusa
09/23/2025, 2:44 AMMarvin
09/23/2025, 2:44 AMMarvin
09/23/2025, 2:50 AMprefect worker start
on always‑on compute (EC2, an ECS service, Kubernetes, or a Dockerized service on a server) so it keeps polling your work pool when your machine is off.
Also, small correction: ECR is the image registry; Prefect’s pool type is ECS. Did you mean an ECS work pool?
Good options to keep your worker alive
- EC2 instance (systemd service)
- Pros: simple, inexpensive, easy to manage
- You’ll need Prefect + AWS credentials on the instance
- Example systemd unit:
# /etc/systemd/system/prefect-worker.service
[Unit]
Description=Prefect Worker for ECS pool
After=network-online.target
[Service]
Environment=PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>>
Environment=PREFECT_API_KEY=<your-api-key>
# Provide AWS creds via instance role (preferred) or env vars if needed
ExecStart=/usr/local/bin/prefect worker start -p my-ecs-pool -n ec2-worker-1
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
Then:
sudo systemctl daemon-reload
sudo systemctl enable --now prefect-worker
- Docker on a server (with restart policy)
- Pros: easy to ship/run; Docker restarts it on reboot
- Example compose:
services:
worker:
image: prefecthq/prefect:3
command: prefect worker start -p my-ecs-pool -n docker-worker-1
environment:
PREFECT_API_URL: <https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>>
PREFECT_API_KEY: <your-api-key>
# If not using instance/task roles:
# AWS_REGION: <region>
# AWS_ACCESS_KEY_ID: <key>
# AWS_SECRET_ACCESS_KEY: <secret>
restart: unless-stopped
- ECS service (Fargate or EC2)
- Build a small container whose entrypoint is:
prefect worker start -p my-ecs-pool -n ecs-service-worker
- Push it to ECR, then run it as an ECS Service with desired count 1 and a task role that allows ECS run-task, logs, etc. This keeps a worker alive in ECS itself.
- Kubernetes
- Run a Deployment with a Prefect image and the command above; Kubernetes will restart the pod if it dies.
Key notes
- In all cases, configure your Prefect connection via PREFECT_API_URL
and PREFECT_API_KEY
, and give the worker AWS permissions (ideally via an instance or task role) so it can start ECS tasks for your flow runs.
- Command you’ll run in these environments:
prefect worker start -p my-ecs-pool
You can add -n <name>
to identify the worker.
Alternative: avoid a worker entirely
- If you’d prefer not to run a long‑lived worker, you can use a serverless/push work pool type. Push pools do not require a worker; Prefect triggers your runs directly in the provider. If that’s appealing, I can help you pick the right push pool for AWS and set it up.
Helpful docs
- ECS work pool guide: Prefect AWS ECS guide
- Prefect settings (API URL and key): Manage Prefect settings
- Running long‑lived processes: Daemonize processes
- Serverless/push pools overview: Serverless (push) work pools
A couple of quick questions so I can tailor exact steps:
- Can you confirm you’re using an ECS work pool (not ECR)?
- Where would you prefer to host the worker: EC2, ECS service, Kubernetes, or a Dockerized server?
- Are you using Prefect Cloud, and do you already have an API key handy?Fugo Takefusa
09/23/2025, 3:13 AMMarvin
09/23/2025, 3:14 AMMarvin
09/23/2025, 3:15 AMprefect worker start
on always‑on compute, not your laptop. For an ECS work pool, good options are:
- EC2 instance (as a systemd service)
- Keeps the worker running across reboots and restarts.
- Provide Prefect API settings and AWS permissions (ideally via an instance role).
- Example:
# /etc/systemd/system/prefect-worker.service
[Unit]
Description=Prefect Worker for ECS pool
After=network-online.target
[Service]
Environment=PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>>
Environment=PREFECT_API_KEY=<your-api-key>
ExecStart=/usr/local/bin/prefect worker start -p my-ecs-pool -n ec2-worker-1
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
Then:
sudo systemctl daemon-reload
sudo systemctl enable --now prefect-worker
- Docker on a server (with restart policy)
- Easy to run anywhere you have Docker; Docker restarts it on reboot/crash.
- Example:
services:
worker:
image: prefecthq/prefect:3
command: prefect worker start -p my-ecs-pool -n docker-worker-1
environment:
PREFECT_API_URL: <https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>>
PREFECT_API_KEY: <your-api-key>
# Prefer instance/task roles for AWS; env vars only if needed:
# AWS_REGION: <region>
# AWS_ACCESS_KEY_ID: <key>
# AWS_SECRET_ACCESS_KEY: <secret>
restart: unless-stopped
- ECS service (runs the worker inside ECS)
- Build or use a Prefect image and set the command to:
prefect worker start -p my-ecs-pool -n ecs-service-worker
- Run it as an ECS Service (desired count 1) with a task role that can create ECS tasks, write logs, etc. This keeps the worker alive inside ECS itself.
- Kubernetes
- Run a Deployment that starts the worker; K8s will keep it healthy.
Notes
- Make sure the worker can reach Prefect Cloud (PREFECT_API_URL
and PREFECT_API_KEY
) and has AWS permissions (instance/task role recommended).
- Command you’ll run in these environments:
prefect worker start -p my-ecs-pool
Optionally add -n <name>
and --work-queue <queue>
.
If you’d prefer not to run a long‑lived worker at all, consider switching to a serverless/push work pool type for AWS; push pools don’t require a worker.
Helpful docs
- ECS work pool guide: Prefect AWS ECS guide
- Prefect settings (API URL and key): Manage Prefect settings
- Daemonizing workers (systemd, etc.): Daemonize processes
- Serverless/push pools overview: Serverless (push) work pools
Want me to tailor an exact setup? Tell me which target you prefer (EC2, ECS service, Docker on a server, or Kubernetes) and I’ll provide the step‑by‑step.