<@ULVA73B9P> is it possible to have multiple worke...
# ask-marvin
m
@Marvin is it possible to have multiple workers under one woker pool, so under one ecs worker pool two workera with different memory configurations
m
thought for 274.4 seconds
Short answer: Yes—you can run multiple workers under a single ECS work pool. However, CPU/memory are not set on the worker itself; they’re defined by the ECS job configuration from the work pool’s base job template and can be overridden per-deployment. So you can have two deployments in the same pool with different CPU/memory, and any number of workers can service them. How to do it - Create one ECS work pool. - Define separate deployments (e.g., “small” and “large”) with different ECS job_variables (cpu/memory). - Optionally use separate work queues and point different workers at different queues if you want isolation. - Start multiple workers that poll the same pool (and optionally specific queues). Example: two sizes in one pool using EC2 launch type prefect.yaml
Copy code
deployments:
  - name: my-flow-small
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-ecs-pool
    job_variables:
      cluster: arn:aws:ecs:ap-southeast-1:123456789012:cluster/my-ecs-cluster
      launch_type: EC2
      cpu: 256          # 0.25 vCPU
      memory: 512       # MiB
      subnets: ["subnet-abc123"]
      security_groups: ["sg-abc123"]
      assign_public_ip: false
      task_role_arn: arn:aws:iam::123456789012:role/my-ecs-task-role
      execution_role_arn: arn:aws:iam::123456789012:role/my-ecs-exec-role
      # container overrides
      env:
        PREFECT_API_URL: http://<your-ec2-ip>:4200/api
      command: []       # leave empty to use flow's default

  - name: my-flow-large
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-ecs-pool
    job_variables:
      cluster: arn:aws:ecs:ap-southeast-1:123456789012:cluster/my-ecs-cluster
      launch_type: EC2
      cpu: 1024         # 1 vCPU
      memory: 2048      # MiB
      subnets: ["subnet-abc123"]
      security_groups: ["sg-abc123"]
      assign_public_ip: false
      task_role_arn: arn:aws:iam::123456789012:role/my-ecs-task-role
      execution_role_arn: arn:aws:iam::123456789012:role/my-ecs-exec-role
      env:
        PREFECT_API_URL: http://<your-ec2-ip>:4200/api
      command: []
Start multiple workers for the same pool (optionally pinned to different queues):
Copy code
prefect worker start -p my-ecs-pool -n worker-small --work-queue small
prefect worker start -p my-ecs-pool -n worker-large --work-queue large
Notes for your setup (ECS EC2 on a single host) - Ensure your requested cpu/memory combinations are supported by your EC2 container instance and ECS task definition constraints. - If you’re using capacity providers for EC2, set a capacity_provider_strategy instead of launch_type, otherwise keep launch_type: EC2. - For clean separation (e.g., small vs large, or dev/stage/prod), use work queues and target workers to specific queues. Docs - ECS guide: Prefect ECS guide - Customizing job variables: Customize job variables - Variables concept (for templating and overrides): Variables in Prefect 3 If you want, I can sketch a pool base job template JSON for ECS EC2 as a starting point and show how to wire queues for small/large.
m
@Marvin if a flow fail due to low memory of the worker can i auto re run with larger worker
m
🔄 Researching your answer... this may take a while 🔧 Using tool:
research_prefect_topic
📊 Tools used: •
check_cli_command
(2x) •
research_prefect_topic
(1x)