Great question — you can absolutely run Prefect in a hybrid setup on AWS ECS. Below are the end-to-end steps for Prefect 3.x, from prerequisites through your first run. Assumptions - You’re using Prefect 3.x (hybrid execution: Prefect Cloud orchestrates; ECS runs your code in your AWS account) - You have an AWS account and permissions to create ECS tasks, IAM roles, VPC networking, and push images to ECR 1) Prerequisites in AWS - Networking: a VPC with subnets and security groups your ECS tasks can use - ECS cluster: create one (Fargate recommended for simplicity) - IAM roles: - Execution role: allows ECS agent to pull from ECR and write logs to CloudWatch (e.g., AWS managed policy AmazonECSTaskExecutionRolePolicy) - Task role: permissions your flow needs (S3, Secrets Manager, etc.) - ECR repository: where you will push your container image - CloudWatch Logs (optional but recommended): a log group for ECS task logs 2) Install Prefect with AWS extras - Either install the extras in your environment:

pip install "prefect[aws]"

- Or, when running CLI commands, use uv so the extras are available at runtime:

uv run --with prefect[aws] prefect version

3) Create an ECS work pool (hybrid) You can do this in the UI or CLI: - UI: Create a Work Pool and choose AWS ECS (hybrid) - CLI:

prefect work-pool create "my-ecs-pool" --type ecs

4) Get and customize the ECS job template (defaults) - Inspect the default base job template for ECS to understand available variables:

prefect work-pool get-default-base-job-template --type ecs > ecs-template.json

- Edit ecs-template.json to set sane defaults you’ll reuse (cluster, launch_type, CPU/memory, container_name, VPC/networking, CloudWatch logs, roles). Then apply it to your work pool:

prefect work-pool create "my-ecs-pool" --type ecs --base-job-template ecs-template.json --overwrite

Examples of commonly used variables in job templates: - cluster, launch_type ("FARGATE" or "EC2") - container_name, image, cpu, memory - task_role_arn, execution_role_arn - network_configuration: - awsvpcConfiguration: subnets, securityGroups, assignPublicIp ("ENABLED" or "DISABLED") - configure_cloudwatch_logs, cloudwatch_logs_prefix - task_definition_arn (if using a pre-registered task definition) - task_definition (template) if you want Prefect to register the task definition for you 5) Build and push your image to ECR - Make sure your flow’s runtime and dependencies are in an image and push it to ECR:

# Example (replace with your account/region/repo)
docker build -t <http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest> .
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <http://123456789012.dkr.ecr.us-east-1.amazonaws.com|123456789012.dkr.ecr.us-east-1.amazonaws.com>
docker push <http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest>

- Alternatively, you can let Prefect build/push via DockerImage when you deploy (ensure your local Docker is logged into ECR). 6) Create a deployment that targets the ECS pool Option A — Pythonic deploy from source: ``` from prefect import flow, get_run_logger from prefect.flows import Flow # Optional: from prefect.docker import DockerImage @flow def hello(): get_run_logger().info("Hello from ECS") # Use your Git repo or local source. Example uses GitHub: Flow.from_source( source="https://github.com/your-org/your-repo", entrypoint="flows/hello.py:hello", ).deploy( name="hello-ecs", work_pool_name="my-ecs-pool", # If you already built/pushed the image: image="123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest", build=False, push=False, # ECS-specific overrides (override pool defaults as needed) job_variables={ "cluster": "my-ecs-cluster", "launch_type": "FARGATE", "container_name": "flow", "cpu": 1024, "memory": 2048, "network_configuration": { "awsvpcConfiguration": {

"subnets": ["subnet-abc123", "subnet-def456"], "securityGroups": ["sg-0123456789abcdef0"], "assignPublicIp": "ENABLED" } }, # OPTION 1: Use an existing Task Definition # "task_definition_arn": "arnawsecsus east 1123456789012task definition/prefect flows1", # OPTION 2: Ask Prefect to register a task definition from this template "task_definition": { "family": "prefect-flows", "cpu": "1024", "memory": "2048", "executionRoleArn": "arnawsiam:123456789012role/ecsExecutionRole", "containerDefinitions": [ {"name": "flow", "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest"} ], }, "configure_cloudwatch_logs": True, "cloudwatch_logs_prefix": "prefect/flows" }, )

Option B — CLI (prefect.yaml)
- Define your deployment in prefect.yaml with work_pool and job_variables, then:

prefect deploy

7) Start the ECS worker (hybrid polling)
Run a worker anywhere that has:
- Access to Prefect Cloud API
- AWS credentials to call ECS APIs
- The prefect_aws collection installed

Set credentials and start:

export PREFECT_API_URL="https://api.prefect.cloud/api/accounts/&lt;acct&gt;/workspaces/&lt;ws&gt;" export PREFECT_API_KEY="<your-prefect-api-key>" # AWS auth (choose one) export AWS_PROFILE="my-aws-profile" # or export AWS_ACCESS_KEY_ID="..." export AWS_SECRET_ACCESS_KEY="..." export AWS_DEFAULT_REGION="us-east-1" uv run --with prefect[aws] prefect worker start -p my-ecs-pool

8) Run and observe
- Trigger a run from the UI or with:

prefect deployment run "hello/hello-ecs" ``` - Watch logs in the Prefect UI and/or CloudWatch Tips and common gotchas - Image and container_name must match what your task definition expects - For Fargate, use awsvpc networking and supply VPC subnets and security groups - If using an existing task definition ARN, you do not need to supply the task_definition template - CloudWatch logging: set configure_cloudwatch_logs=true and a prefix, and ensure your execution role has logging permissions - IAM: the execution role needs ECR pull and CloudWatch logs; the task role needs whatever your flow code uses (S3, Secrets Manager, etc.) Docs - ECS worker guide (Prefect 3.x): AWS ECS + Prefect guide Want me to tailor a ready-to-run template? - Let me know: - ECS cluster name and whether you use Fargate or EC2 - Your VPC subnet IDs and security group IDs - Whether you’ll use an existing Task Definition ARN or have Prefect register one - Your ECR image URI I’ll generate a minimal ecs-template.json and a matching deployment snippet for you.

Got it — since you already created the ECS work pool in the Prefect UI, here’s exactly what to put in its configuration, what you need on the AWS side, and a working sample ECS Fargate task definition. High-level: in Prefect 3.x hybrid ECS, the worker submits an ECS task that runs your flow. The worker will override the container’s command and inject the necessary Prefect environment variables. Most runtime details can be set as Work Pool “job variables.” What to configure in your ECS work pool (job variables) Paste values like these into your Work Pool’s “Default Job Variables” (UI) and adjust for your account:

{
  "cluster": "my-ecs-cluster",                  // ECS cluster name or ARN
  "launch_type": "FARGATE",                     // or use capacityProviderStrategy
  "container_name": "flow",                     // MUST match the container name in your task definition
  "image": "<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest>",

  // CPU and memory used for overrides and/or task def (ensure they align)
  "cpu": 1024,
  "memory": 2048,

  // If you are using a pre-registered task definition (recommended):
  "task_definition_arn": "arn:aws:ecs:us-east-1:123456789012:task-definition/prefect-flow:1",

  // Networking (Fargate requires awsvpc)
  "network_configuration": {
    "awsvpcConfiguration": {
      "subnets": ["subnet-abc123", "subnet-def456"],
      "securityGroups": ["sg-0123456789abcdef0"],
      "assignPublicIp": "ENABLED"   // enable if using public subnets or no NAT
    }
  },

  // Roles (required if Prefect will register task definitions for you; also useful as overrides)
  "execution_role_arn": "arn:aws:iam::123456789012:role/ecsExecutionRole",
  "task_role_arn": "arn:aws:iam::123456789012:role/YourTaskRole",

  // Logging (recommended)
  "configure_cloudwatch_logs": true,
  "cloudwatch_logs_prefix": "prefect/flows",

  // Quality of life
  "stream_output": true
}

Notes: - container_name must match the container name in your task definition. - Choose one: task_definition_arn (pre-registered) OR task_definition (template). If you provide both, the ARN is used. - For Fargate, make sure your CPU/memory are valid Fargate sizes and consistent between overrides and task definition. AWS requirements (checklist) - ECS Cluster: available and in the correct region. - Networking: VPC subnets and security groups. For Fargate you must use awsvpc networking. - If your tasks need internet (to reach Prefect Cloud API, ECR, PyPI, etc.), provide outbound access via Public IP (public subnets) or via NAT gateway (private subnets). - ECR repo + image: your flow image is pushed to ECR; execution role can pull it. - IAM roles: - Execution role (ecsTaskExecutionRole) with AmazonECSTaskExecutionRolePolicy and permissions to read from your ECR repo and write CloudWatch Logs. - Task role: whatever your flow code needs (e.g., S3, Secrets Manager). - CloudWatch Logs: log group exists or allow the logs driver to create it. - ECS task definition: either supply an ARN (recommended) or let Prefect register from a template at run time. Sample ECS Fargate task definition (works well with Prefect) Register this in AWS ECS, then reference its ARN in your work pool’s job variables as shown above. ``` { "family": "prefect-flow", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "1024", "memory": "2048", "executionRoleArn": "arnawsiam:123456789012role/ecsExecutionRole", "taskRoleArn": "arnawsiam:123456789012role/YourTaskRole", "runtimePlatform": { "operatingSystemFamily": "LINUX" }, "containerDefinitions": [ { "name": "flow", // IMPORTANT: matches job variable: container_name "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest", "essential": true, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-region": "us-east-1",

"awslogs-group": "/ecs/prefect/flows", "awslogs-stream-prefix": "prefect", "awslogs-create-group": "true" } } // Do NOT hardcode "command" here; Prefect worker will override it per flow run // Add env if you want static app env vars; Prefect will inject its own as needed } ] }

Common causes of “error while running the task”
- Container name mismatch: your task definition containerDefinitions[].name must equal the work pool’s container_name (default “flow”).
- Execution vs Task role confusion:
  - executionRoleArn must have ECR pull + CloudWatch Logs
  - taskRoleArn must have app permissions (S3, Secrets Manager, etc.)
- Networking issues:
  - Fargate requires awsvpc; ensure subnets and securityGroups are valid for the cluster and region.
  - If no internet egress, tasks can’t reach Prefect Cloud or ECR; add NAT or assignPublicIp: "ENABLED" in a public subnet.
- CPU/memory mismatch: overrides in job variables must be valid for Fargate sizing and consistent with the task definition.
- Wrong cluster/region/ARN strings: ensure your job variables reference the right cluster and region.
- ECR auth: executionRole must be able to pull the image from your ECR repo.

If you want Prefect to register the task definition for you
Instead of task_definition_arn, put a task_definition template in your job variables. Keep it minimal and let Prefect handle registration per run:

"task_definition": { "family": "prefect-flow", "cpu": "1024", "memory": "2048", "executionRoleArn": "arnawsiam:123456789012role/ecsExecutionRole", "taskRoleArn": "arnawsiam:123456789012role/YourTaskRole", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "containerDefinitions": [ { "name": "flow", "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-ecs:latest", "essential": true } ] } ``` Reference docs - ECS + Prefect guide (3.x): AWS ECS + Prefect guide If you can share the exact ECS error message and your current task definition JSON, I can pinpoint the fix quickly.