<@ULVA73B9P> I'm using the hybrid execution model ...
# ask-marvin
w
@Marvin I'm using the hybrid execution model using AWS ECS. in my prefect.yaml file i have a deployment specified like so:
Copy code
deployments:  
  - name: my_ecs_deployment_dev
    entrypoint: prefect/my_flow.py:my_flow
    job_variables:
      image: "{{ $image_tag }}"
    work_pool:
      name: *dev_work_pool
      image: "{{ $image_tag }}"
However when I deploy this and schedule a run, the image specified in the deployment is not used. when i look in the prefect ui, there are no job variables and my deployment doesn't reference my image name anywhere.
m
thought for 344.4 seconds
It’s not you. It’s the YAML. In Prefect 3.x, job variables must live under the work pool section, and
work_pool.image
isn’t a valid key. Your top-level
job_variables
is also ignored. That’s why the UI shows no job variables and your image isn’t used. Here’s the corrected structure for an ECS work pool deployment:
Copy code
# optional if you want templating
variables:
  image_tag: "<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest>"

deployments:
  - name: my_ecs_deployment_dev
    entrypoint: prefect/my_flow.py:my_flow
    work_pool:
      name: dev_work_pool
      job_variables:
        image: "{{ $image_tag }}"
Notes: -
job_variables
must be nested under
work_pool
. -
image
is the right key for ECS workers. -
{{ $image_tag }}
will only render if you define
image_tag
in the top-level
variables:
block (as above) or via your project variables. Otherwise it renders to nothing, which is… unhelpful. After updating, redeploy from your project root:
Copy code
prefect deploy
Then check the deployment in the UI — you should see the job variables (including the image) on the deployment page. If you still don’t see them: - Confirm the work pool is of type ECS and your worker is running against that pool. - Make sure you deployed from the same project directory containing this prefect.yaml. - Verify the variable actually resolves (try hardcoding the image once to confirm the path). Docs, in case you like reading specifications as much as I do: - Customize job variables: Customize job variables - prefect.yaml reference: prefect.yaml reference - ECS guide: AWS ECS guide If you share how you’re setting
image_tag
and the exact work pool type/name, I’ll sanity-check the rest. I live for this. Obviously.
w
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1074, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images.
@Marvin help with this error: File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1074, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images.
m
thought for 146.6 seconds
That error is coming from ECS, not Prefect. It means your Fargate task definition doesn’t set an execution role, so ECS can’t pull your ECR image. What you need - An execution role (usually named ecsTaskExecutionRole) with AmazonECSTaskExecutionRolePolicy attached. This role is used by ECS to pull images from ECR and write logs. - Optionally, a separate task role for your flow code’s AWS calls. - Your ECS work pool’s base job template must include executionRoleArn (and taskRoleArn if you use it). - Your deployment must pass those ARNs via job variables that match the base job template. Checklist 1) Create/identify the roles - Execution role: attach AmazonECSTaskExecutionRolePolicy. If your ECR repo is in another account or KMS-encrypted, add the needed ECR/KMS permissions. - Task role: permissions your flow actually needs (S3, SNS, etc.). - Ensure the AWS credentials used by your Prefect worker have iam:PassRole permission on both ARNs. 2) Ensure your ECS work pool base job template references these fields In the Prefect UI: Work Pools > your ECS pool > Edit base job template. Make sure it includes placeholders like:
Copy code
{
  "executionRoleArn": "{{ execution_role_arn }}",
  "taskRoleArn": "{{ task_role_arn }}",
  "containerDefinitions": [
    {
      "image": "{{ image }}",
      ...
    }
  ],
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "subnets": {{ subnets }},
      "securityGroups": {{ security_groups }},
      "assignPublicIp": "{{ assign_public_ip }}"
    }
  },
  "cpu": "{{ cpu }}",
  "memory": "{{ memory }}",
  "cluster": "{{ cluster }}"
}
If executionRoleArn isn’t present in the template, Prefect can’t set it and ECS will throw exactly the error you’re seeing. 3) Pass the variables in your deployment
Copy code
variables:
  image_tag: "<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest>"

deployments:
  - name: my_ecs_deployment_dev
    entrypoint: prefect/my_flow.py:my_flow
    work_pool:
      name: dev_work_pool
      job_variables:
        image: "{{ $image_tag }}"
        execution_role_arn: "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
        task_role_arn: "arn:aws:iam::123456789012:role/my-ecs-task-role"
        cluster: "my-ecs-cluster"
        cpu: "512"
        memory: "1024"
        subnets: ["subnet-abc", "subnet-def"]
        security_groups: ["sg-123456"]
        assign_public_ip: "ENABLED"
Then redeploy:
Copy code
prefect deploy
If you’re unsure of the exact variable names, open your work pool’s base job template — those placeholders determine the correct keys. Prefect doesn’t invent them at runtime. If you share your ECS work pool base job template (or a redacted version), I’ll confirm the exact keys you should set. Also confirm your worker’s AWS credentials have iam:PassRole for both roles; without that, ECS can’t use them even if they’re set.