Madushan Pathirana
09/14/2025, 4:03 AMMarvin
09/14/2025, 4:03 AMMarvin
09/14/2025, 4:14 AM{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECSTasksTrust",
"Effect": "Allow",
"Principal": {
"Service": "<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>"
},
"Action": "sts:AssumeRole"
}
]
}
B) ECS worker task role (the role your Prefect ECS Worker container runs as)
- This role must be able to register/run/stop tasks in your ECS cluster and pass the roles to those tasks. The execution role alone is not enough.
Minimal policy (scope Resource to your ARNs where possible):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECSDescribeAndRun",
"Effect": "Allow",
"Action": [
"ecs:RunTask",
"ecs:StopTask",
"ecs:RegisterTaskDefinition",
"ecs:DescribeTasks",
"ecs:DescribeTaskDefinition",
"ecs:DescribeClusters",
"ecs:ListTasks",
"ecs:ListTaskDefinitions",
"ecs:ListClusters"
],
"Resource": "*"
},
{
"Sid": "PassRolesToTasks",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": [
"arn:aws:iam::<your-account-id>:role/<YourEcsTaskRoleForFlows>",
"arn:aws:iam::<your-account-id>:role/<YourEcsTaskExecutionRole>"
],
"Condition": {
"StringEquals": {
"iam:PassedToService": "<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>"
}
}
}
]
}
Notes:
- The “task role” that your flow-run containers assume (for S3, DB, etc.) is typically <YourEcsTaskRoleForFlows>. The worker needs iam:PassRole permission on that.
- Private ECR pulls are handled by the execution role via AmazonECSTaskExecutionRolePolicy; the worker role does not need ECR permissions.
- CloudWatch Logs permissions are in the execution role (not the worker role).
Docs: Prefect ECS guide
2) ECS Task Definition for Prefect Server (EC2 launch type)
- Prefect Server listens on 4200 by default and serves both API and UI.
- Use Postgres with async driver: postgresql+asyncpg://...
- The Prefect image may not include asyncpg; safest is to add it. Example Dockerfile:
FROM prefecthq/prefect:3-latest
RUN pip install --no-cache-dir asyncpg
Task definition template (awsvpc, EC2 launch type):
```
{
"family": "prefect-server",
"networkMode": "awsvpc",
"requiresCompatibilities": ["EC2"],
"cpu": "512",
"memory": "1024",
"executionRoleArn": "arnawsiam:<your account id>role/<YourEcsTaskExecutionRole>",
"taskRoleArn": "arnawsiam:<your account id>role/<MinimalServerTaskRoleOrNone>",
"containerDefinitions": [
{
"name": "prefect-server",
"image": "<your-registry>/prefect-server:latest",Marvin
09/14/2025, 4:14 AMNetworking/Security Group tips:
- Attach an ALB/NLB that targets port 4200 on this service.
- Allow inbound 4200 from the ALB SG to the task ENI SG.
- Allow outbound 5432 from the server task SG to the RDS SG; and allow inbound 5432 from the server/worker SGs on the RDS SG.
Docs and refs:
- <https://docs-3.prefect.io/v3/how-to-guides/self-hosted/server-docker|Run Prefect Server in Docker> (same env vars apply in ECS)
- Settings reference (API/UI env vars): <https://docs-3.prefect.io/v3/api-ref/settings-ref|Prefect settings>
3) Configure S3 for results (default results storage)
- Create an S3Bucket block (and AwsCredentials block if you are not using IAM roles) in Prefect.
- Then set `PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK` to your block slug `s3-bucket/<block-name>` in the environments that run your flows (i.e., your worker/flow task definitions), not on the server.
Example to create blocks once:
from prefect_aws.s3 import S3Bucket
from prefect_aws.credentials import AwsCredentials
aws = AwsCredentials(
aws_access_key_id="...",
aws_secret_access_key="...",
region_name="ap-southeast-1"
)
aws.save("my-aws-creds", overwrite=True)
s3 = S3Bucket(
bucket_name="my-prefect-results",
aws_credentials=aws
)
s3.save("default-results", overwrite=True)
Then add this env var to your worker and/or flow-run task definitions:
PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK = "s3-bucket/default-results"
Notes:
- If you use IAM roles for the flow task role instead of static keys, you can omit AwsCredentials and create the S3Bucket block without credentials; the task role must have S3 permissions.
- Install the AWS integration in images that run flows: add `pip install prefect-aws` to your worker/flow image.
4) Worker container notes (if you run the worker in ECS too)
- Image should include `prefect-aws` for ECS and S3 integrations:
FROM prefecthq/prefect:3-latest
RUN pip install --no-cache-dir prefect-aws
- Env vars for the worker container:
PREFECT_API_URL = "https://prefect.yourdomain.com/api"
PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK = "s3-bucket/default-results" # optional global default
- Start the worker with your ECS pool (replace pool name):
prefect worker start --pool "my-ecs-pool"
- The worker’s task role = the “ECS worker task role” policy shown above. The flow-run tasks that it launches can use a separate “flow task role” with S3/RDS/etc. access appropriate to your flows.
5) RDS/Postgres connection reminder
- Prefect Server uses `PREFECT_SERVER_DATABASE_CONNECTION_URL`. Example with RDS:Marvin
09/14/2025, 4:14 AMasyncpg.
6) Quick checklist for your setup in ap-southeast-1
- ECS service (EC2 launch type), awsvpc networking, SGs wired to RDS correctly
- ALB/NLB → target port 4200 → health check /api/health
- Execution role: AmazonECSTaskExecutionRolePolicy
- Worker task role: ECS permissions + iam:PassRole (as above)
- Flow task role (used by the tasks the worker launches): S3 permissions (and any other AWS services your flows use)
- Worker and flow images include `prefect-aws`; server image includes asyncpg
- Clients/workers use PREFECT_API_URL = https://<your-hostname>/api
Questions for you
- When you said “S3 for artifacts,” did you mean default results storage on S3? If you want something different for artifacts specifically, let me know your retention/size goals and we can propose an approach.
- Do you plan to front the server with an ALB on a custom domain? If yes, I’ll tailor `PREFECT_UI_URL`/`PREFECT_API_URL` and SGs accordingly.
- Do you want me to scope the IAM policy to specific cluster/task definition ARNs in your account? If you share ARNs (or a naming convention), I can provide scoped versions.
Helpful docs
- Prefect Server in Docker (env vars/settings apply to ECS)
- Prefect ECS guide
- Settings reference
If you share your cluster name, task family names, and ARNs, I can generate fully-scoped IAM JSON and a final task definition you can register as-is.Madushan Pathirana
09/14/2025, 5:00 AMMarvin
09/14/2025, 5:00 AMMarvin
09/14/2025, 5:00 AM"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
"subnet-xxxxxxxx" // public subnet for Option A (or private subnet for Option B)
],
"securityGroups": [
"sg-xxxxxxxx"
],
"assignPublicIp": "ENABLED" // Option A only. Use "DISABLED" for Option B
}
}
What to set for Prefect settings
- Server container:
- PREFECT_SERVER_DATABASE_CONNECTION_URL = postgresql+asyncpg//<user><pass>@<rds-endpoint>:5432/<db>
- Optional: PREFECT_UI_URL = http//<public ip>4200 (Option A). If you don’t set it, the UI still works; this helps links render nicely.
- Worker/clients:
- PREFECT_API_URL = http//<public ip>4200/api (Option A) or http//<private ip>4200/api (Option B)
- If you’re using S3 for results by default: PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK = s3-bucket/<your-s3-block-name>
Networking/Security quick tips
- RDS SG: allow inbound 5432 from the Prefect Server task SG and from any flow-run task SGs that need DB access.
- ECS (no LB) health checks: rely on the container healthCheck you put in the task definition. ECS can restart unhealthy containers without an ALB.
When you’re ready, adding an ALB later gives you:
- Stable DNS, TLS/HTTPS, and zero-downtime deploys
- No IP-changes for clients/workers
- Simpler UI access via a hostname
Docs for reference
- Prefect Server in containers: Prefect Server in Docker
- Prefect settings reference: Settings reference
- Prefect ECS guide: Prefect ECS guide
Want me to tailor this to Option A (public IP) or Option B (private + tunnel) for your exact subnets/SGs and spit out a ready-to-run create-service JSON? If you share your subnet IDs and SG names, I’ll generate it.