<@ULVA73B9P> I need to host prefect server in ecs....
# ask-marvin
m
@Marvin I need to host prefect server in ecs. I need the ecs task execution role permissions required , which i will use for the ecs worker too. also i need ecs task definition for prefect server , i need to add postgress db and s3 for artifacts
m
thought for 655.7 seconds
Got it — here’s a practical, copy/paste-ready setup to run Prefect Server on ECS (EC2 launch type) with Postgres (RDS) and S3-backed results. I’m assuming Prefect 3.x. Summary of what’s included below: - IAM roles/policies you need (execution role + worker task role) - ECS Task Definition JSON for Prefect Server (EC2 launch type) with RDS connection - How to configure S3 for results (the “artifacts” you mentioned — see note below) - Key env vars and image notes - Helpful links Important note on “artifacts” vs “results” - Prefect 3 “artifacts” (markdown, tables, images) are stored in the Prefect database by default. - If you meant storing flow results on S3, that’s configured via a Results Storage Block and an env var (details below). If you really want artifacts in S3, we can explore patterns, but it’s not a built-in storage backend for artifacts. - I’ll proceed assuming you mean S3 for default results storage. 1) IAM: roles and policies You need two IAM roles: A) ECS task execution role (used by ECS agent to pull images and write logs) - Attach AWS managed policy: AmazonECSTaskExecutionRolePolicy - If you pull images from private ECR in another account or use Secrets Manager/SSM for env vars, add those read permissions as well. Trust policy (ecs-tasks.amazonaws.com):
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ECSTasksTrust",
      "Effect": "Allow",
      "Principal": {
        "Service": "<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
B) ECS worker task role (the role your Prefect ECS Worker container runs as) - This role must be able to register/run/stop tasks in your ECS cluster and pass the roles to those tasks. The execution role alone is not enough. Minimal policy (scope Resource to your ARNs where possible):
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ECSDescribeAndRun",
      "Effect": "Allow",
      "Action": [
        "ecs:RunTask",
        "ecs:StopTask",
        "ecs:RegisterTaskDefinition",
        "ecs:DescribeTasks",
        "ecs:DescribeTaskDefinition",
        "ecs:DescribeClusters",
        "ecs:ListTasks",
        "ecs:ListTaskDefinitions",
        "ecs:ListClusters"
      ],
      "Resource": "*"
    },
    {
      "Sid": "PassRolesToTasks",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": [
        "arn:aws:iam::<your-account-id>:role/<YourEcsTaskRoleForFlows>",
        "arn:aws:iam::<your-account-id>:role/<YourEcsTaskExecutionRole>"
      ],
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>"
        }
      }
    }
  ]
}
Notes: - The “task role” that your flow-run containers assume (for S3, DB, etc.) is typically <YourEcsTaskRoleForFlows>. The worker needs iam:PassRole permission on that. - Private ECR pulls are handled by the execution role via AmazonECSTaskExecutionRolePolicy; the worker role does not need ECR permissions. - CloudWatch Logs permissions are in the execution role (not the worker role). Docs: Prefect ECS guide 2) ECS Task Definition for Prefect Server (EC2 launch type) - Prefect Server listens on 4200 by default and serves both API and UI. - Use Postgres with async driver:
postgresql+asyncpg://...
- The Prefect image may not include asyncpg; safest is to add it. Example Dockerfile:
Copy code
FROM prefecthq/prefect:3-latest
RUN pip install --no-cache-dir asyncpg
Task definition template (awsvpc, EC2 launch type): ``` { "family": "prefect-server", "networkMode": "awsvpc", "requiresCompatibilities": ["EC2"], "cpu": "512", "memory": "1024", "executionRoleArn": "arnawsiam:&lt;your account id&gt;role/<YourEcsTaskExecutionRole>", "taskRoleArn": "arnawsiam:&lt;your account id&gt;role/<MinimalServerTaskRoleOrNone>", "containerDefinitions": [ { "name": "prefect-server", "image": "<your-registry>/prefect-server:latest",
"essential": true, "command": [ "prefect", "server", "start", "--host", "0.0.0.0", "--port", "4200", "--ui" ], "environment": [ { "name": "PREFECT_SERVER_DATABASE_CONNECTION_URL", "value": "postgresql+asyncpg//&lt;db user&gt;<db_password>@<rds-endpoint>:5432/<db_name>" }, { "name": "PREFECT_UI_URL", "value": "https://prefect.yourdomain.com" } // You generally don't need PREFECT_API_URL inside the server container. // Clients and workers should set PREFECT_API_URL to "https://prefect.yourdomain.com/api" ], "portMappings": [ { "containerPort": 4200, "hostPort": 4200, "protocol": "tcp" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/prefect-server", "awslogs-region": "ap-southeast-1", "awslogs-stream-prefix": "ecs" } }, "healthCheck": { "command": [ "CMD-SHELL", "curl -f http://localhost:4200/api/health || exit 1" ], "interval": 30, "timeout": 5, "retries": 3, "startPeriod": 30 } } ] }
Copy code
Networking/Security Group tips:
- Attach an ALB/NLB that targets port 4200 on this service.
- Allow inbound 4200 from the ALB SG to the task ENI SG.
- Allow outbound 5432 from the server task SG to the RDS SG; and allow inbound 5432 from the server/worker SGs on the RDS SG.

Docs and refs:
- <https://docs-3.prefect.io/v3/how-to-guides/self-hosted/server-docker|Run Prefect Server in Docker> (same env vars apply in ECS)
- Settings reference (API/UI env vars): <https://docs-3.prefect.io/v3/api-ref/settings-ref|Prefect settings>

3) Configure S3 for results (default results storage)

- Create an S3Bucket block (and AwsCredentials block if you are not using IAM roles) in Prefect.
- Then set `PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK` to your block slug `s3-bucket/<block-name>` in the environments that run your flows (i.e., your worker/flow task definitions), not on the server.

Example to create blocks once:
from prefect_aws.s3 import S3Bucket from prefect_aws.credentials import AwsCredentials aws = AwsCredentials( aws_access_key_id="...", aws_secret_access_key="...", region_name="ap-southeast-1" ) aws.save("my-aws-creds", overwrite=True) s3 = S3Bucket( bucket_name="my-prefect-results", aws_credentials=aws ) s3.save("default-results", overwrite=True)
Copy code
Then add this env var to your worker and/or flow-run task definitions:
PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK = "s3-bucket/default-results"
Copy code
Notes:
- If you use IAM roles for the flow task role instead of static keys, you can omit AwsCredentials and create the S3Bucket block without credentials; the task role must have S3 permissions.
- Install the AWS integration in images that run flows: add `pip install prefect-aws` to your worker/flow image.

4) Worker container notes (if you run the worker in ECS too)

- Image should include `prefect-aws` for ECS and S3 integrations:
FROM prefecthq/prefect:3-latest RUN pip install --no-cache-dir prefect-aws
Copy code
- Env vars for the worker container:
PREFECT_API_URL = "https://prefect.yourdomain.com/api" PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK = "s3-bucket/default-results" # optional global default
Copy code
- Start the worker with your ECS pool (replace pool name):
prefect worker start --pool "my-ecs-pool"
Copy code
- The worker’s task role = the “ECS worker task role” policy shown above. The flow-run tasks that it launches can use a separate “flow task role” with S3/RDS/etc. access appropriate to your flows.

5) RDS/Postgres connection reminder

- Prefect Server uses `PREFECT_SERVER_DATABASE_CONNECTION_URL`. Example with RDS:
postgresql+asyncpg://prefect_user:strongpassword@prefect-rds.cluster-abcdefghij.ap-southeast-1.rds.amazonaws.com:5432/prefect ``` - Ensure the server image contains
asyncpg
. 6) Quick checklist for your setup in ap-southeast-1 - ECS service (EC2 launch type), awsvpc networking, SGs wired to RDS correctly - ALB/NLB → target port 4200 → health check /api/health - Execution role: AmazonECSTaskExecutionRolePolicy - Worker task role: ECS permissions + iam:PassRole (as above) - Flow task role (used by the tasks the worker launches): S3 permissions (and any other AWS services your flows use) - Worker and flow images include `prefect-aws`; server image includes
asyncpg
- Clients/workers use
PREFECT_API_URL
= https://<your-hostname>/api Questions for you - When you said “S3 for artifacts,” did you mean default results storage on S3? If you want something different for artifacts specifically, let me know your retention/size goals and we can propose an approach. - Do you plan to front the server with an ALB on a custom domain? If yes, I’ll tailor `PREFECT_UI_URL`/`PREFECT_API_URL` and SGs accordingly. - Do you want me to scope the IAM policy to specific cluster/task definition ARNs in your account? If you share ARNs (or a naming convention), I can provide scoped versions. Helpful docs - Prefect Server in Docker (env vars/settings apply to ECS) - Prefect ECS guide - Settings reference If you share your cluster name, task family names, and ARNs, I can generate fully-scoped IAM JSON and a final task definition you can register as-is.
m
@Marvin can i run this without ALB initially , i am new to ecs so i want to keep it simple
m
thought for 31.3 seconds
Yes — you can start without an ALB to keep things simple. Two simple ways to do it Option A: Public IP on the ECS task (no load balancer) - Put the ECS service in a public subnet (has a route to an Internet Gateway). - Enable “Auto-assign public IP” on the service’s networking config (awsvpc mode). - Security group for the task: - Inbound: TCP 4200 from your IP (or your office’s CIDR) - Outbound: allow 5432 to your RDS SG and general egress as needed - Grab the task’s public IP from the ECS console and open http//&lt;public ip&gt;4200 - Caveats: - The public IP changes on each deployment/replace. - No TLS/HTTPS unless you add a proxy later. - Scaling/HA and a stable DNS name are easier with a load balancer (you can add ALB later). Option B: Keep it private and tunnel for UI access (no load balancer) - Run the ECS service in private subnets with no public IP. - Workers/flows in the same VPC talk to Prefect Server via its private IP (PREFECT_API_URL like http//&lt;server private ip&gt;4200/api). - For your browser access to the UI, use: - SSH tunnel via a bastion/EC2, or - AWS Systems Manager Session Manager port forwarding - Caveats: - Still no static address; the private IP changes when the task is replaced. - You’ll update clients/workers (or your tunnel target) when the IP changes, unless you add Service Discovery or an LB later. Minimal changes to your Task Definition - You don’t need to change the container definition. The difference is in the ECS Service networking:
Copy code
"networkConfiguration": {
  "awsvpcConfiguration": {
    "subnets": [
      "subnet-xxxxxxxx"   // public subnet for Option A (or private subnet for Option B)
    ],
    "securityGroups": [
      "sg-xxxxxxxx"
    ],
    "assignPublicIp": "ENABLED"   // Option A only. Use "DISABLED" for Option B
  }
}
What to set for Prefect settings - Server container: - PREFECT_SERVER_DATABASE_CONNECTION_URL = postgresql+asyncpg//&lt;user&gt;<pass>@<rds-endpoint>:5432/<db> - Optional: PREFECT_UI_URL = http//&lt;public ip&gt;4200 (Option A). If you don’t set it, the UI still works; this helps links render nicely. - Worker/clients: - PREFECT_API_URL = http//&lt;public ip&gt;4200/api (Option A) or http//&lt;private ip&gt;4200/api (Option B) - If you’re using S3 for results by default: PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK = s3-bucket/<your-s3-block-name> Networking/Security quick tips - RDS SG: allow inbound 5432 from the Prefect Server task SG and from any flow-run task SGs that need DB access. - ECS (no LB) health checks: rely on the container healthCheck you put in the task definition. ECS can restart unhealthy containers without an ALB. When you’re ready, adding an ALB later gives you: - Stable DNS, TLS/HTTPS, and zero-downtime deploys - No IP-changes for clients/workers - Simpler UI access via a hostname Docs for reference - Prefect Server in containers: Prefect Server in Docker - Prefect settings reference: Settings reference - Prefect ECS guide: Prefect ECS guide Want me to tailor this to Option A (public IP) or Option B (private + tunnel) for your exact subnets/SGs and spit out a ready-to-run create-service JSON? If you share your subnet IDs and SG names, I’ll generate it.