<@ULVA73B9P> When looking at my ECS Work pool on t...
# ask-marvin
t
@Marvin When looking at my ECS Work pool on the Prefect Cloud UI, why does the worker say the "Prefect Version" is 3.4.15 when the task definition for my ECS cluster uses this docker image?:
Copy code
prefecthq/prefect:2.14.13-python3.10
m
thought for 428.1 seconds
Short answer: that “Prefect Version” value on the worker is the version of Prefect installed in the worker process itself — not the version inside the runtime image used for your ECS tasks. - The worker you have connected to the ECS work pool is running Prefect 3.4.15, so the UI shows 3.4.15 for the worker. - Your flow runs, however, are launched into the ECS task definition/image you’ve specified (
prefecthq/prefect:2.14.13-python3.10
). That image’s Prefect version is independent from the worker’s. This mismatch can happen (and can work), because: - The worker’s job is to poll the work pool and provision/run infrastructure (ECS tasks) according to your pool/template/deployment settings. - The flow actually executes inside the container image you specify for the ECS task. That image can have a different Prefect version than the worker. That said, for consistency and access to 3.x features, we generally recommend aligning the runtime image with 3.x as well (for example,
prefecthq/prefect:3.4.15-python3.10
or a
3.x
tag), and using the 3.x deployment patterns. If you stick with a 2.x image, some 3.x features and patterns (e.g., deployment APIs) won’t be available inside that container. Helpful references: - Workers and work pools overview (what the worker does vs. where the flow runs): Blog: Work pools and workers and Upgrade agents to workers (3.x) - Managing deployments (where you can control the runtime image via your pool/job template or deployment config): prefect.yaml and deployments If you want, share how you’re configuring the ECS work pool (base job template or task definition ARN) and I can point you to the exact field to change to upgrade the runtime image.
Your message was too long, here's your message at the allowed limit: Sure here is the yaml file that is used to configure and start up the ecs cluster ```AWSTemplateFormatVersion '2010-09-09' Description Prefect 2 Cluster Roles and Service Infrastructure Parameters envUpper Type String Description The name of the environment to be used as a part of the prefix for all resource names AllowedValues [ DEV PROD ] environment Type String Description The name of the environment to be used as a part of the prefix for all resource names AllowedValues [ dev prod ] cpu Type String Description Allow Dynamic CPU configuration Default 512 AllowedValues [256 512 1024 2048 4096] memory Type String Description Allow Increasing Memory - from 8192 on requires 4096 CPU and increases in 1024 increments default Default 1024 AllowedValues [512 1024 2048 4096 5120 6144 7168 8192 9216 10240] awsaccountid Type String Description AWS Account ID Default xxxx region Type String Description AWS region name Default us-east-2 image Type String Description Docker image from prefect which the agent will use - NOT OUR PY310-base image Default prefecthq/prefect 2 14 13-python3 10 InstanceType Type String Description Choosing t3 micro because it is free Default t3 micro PrefectECSServiceSubnet Type String Description Subnet ID Default subnet-0056ad26401f64636 ECSClusterName Type String Default dev-orion-prefect-2 SecurityGroupIDs Type CommaDelimitedList Default sg-336ce346 SubnetIDs Type CommaDelimitedList Description Subnets in the main VPC to use Default >- subnet-abc123 subnet-abc234 subnet-abc345 VpcID Type String Description Main VPC ID Default vpc-123 #Dict to reference Mappings EnvConfig dev PrefectApiUrl arn aws secretsmanager us-east-2 xxxx secret DEV_PREFECT api_url PrefectApiKey arn aws secretsmanager us-east-2 xxxx secret DEV_PREFECT api_key prod PrefectApiUrl arn aws secretsmanager us-east-2 xxxx secret PROD_PREFECT api_url PrefectApiKey arn aws secretsmanager us-east-2 xxxx secret PROD_PREFECT api_key Resources PrefectFargateCluster Type AWS ECS Cluster Properties ClusterName !Sub prefect-2-cluster-${environment} PrefectLogGroup Type AWS Logs LogGroup Properties LogGroupName !Ref PrefectFargateCluster RetentionInDays 7 ExecutionRole Type AWS IAM Role Properties RoleName !Sub ${environment}_prefect_ecs_execution_role AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service <http //ecs-tasks amazonaws com|ecs-tasks amazonaws com> Action sts AssumeRole Policies - PolicyName SecretsFromParameterStoreCloudWatch PolicyDocument Version 2012-10-17 Statement - Effect Allow Action - ssm GetParameters - logs CreateLogStream - logs CreateLogGroup - logs PutLogEvents - secretsmanager GetSecretValue - kms Decrypt Resource * - PolicyName SecretsManagerDecryptPermission PolicyDocument Version 2012-10-17 Statement - Effect Allow Action - ssm GetParameters - secretsmanager GetSecretValue - kms Decrypt Resource - !Sub arn aws secretsmanager us-east-2 xxxx secret ${envUpper}_PREFECT* - arn aws secretsmanager us-east-2 xxxx secret Docker ManagedPolicyArns - arn aws iam aws policy/service-role/AmazonECSTaskExecutionRolePolicy TaskRole Type AWS IAM Role Properties #Change to lower? RoleName !Sub ${environment}_prefect_ecs_task_role AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service <http //ecs-tasks amazonaws com|ecs-tasks amazonaws com> Action sts AssumeRole Policies - PolicyName PrefectRequirements PolicyDocument Version 2012-10-17 Statement - Effect Allow Action - s3 * - ecr BatchCheckLayerAvailability - ecr BatchGetImage - ecr GetDownloadUrlForLayer - ecr GetAuthorizationToken - ecs RegisterTaskDefinition - ecs DeregisterTaskDefinition - ecs DescribeTaskDefinition - ecs DescribeTasks - ecs RunTask - logs GetLogEvents - ec2 DescribeSubnets - ec2 DescribeVpcs - ecs StopTask - iam PassRole - secretsmanager GetSecretValue - kms Decrypt Resource * PrefectTaskDefinition Type AWS ECS TaskDefinition Properties Family
t
@Marvin Here is the yaml file for the ECS cluster.
m
thought for 328.5 seconds
Thanks! I don’t see the YAML content in your message—could you paste it here in a code block so I can call out exactly where the image is set? In the meantime, here’s how image selection typically works for an ECS work pool: - If your ECS work pool uses a task definition ARN: the image comes from that task definition in AWS. To change the runtime Prefect version, update the task definition to a new revision with your desired image tag (or switch the ARN the pool uses). The worker version in the UI will still reflect the worker process version (3.4.15). - If your ECS work pool does not use a task definition ARN: the image is set in the pool’s base job template under task_definition.containerDefinitions[0].image. You can view your pool’s current template with:
Copy code
prefect work-pool inspect "<your-ecs-pool-name>"
For reference, the default ECS base job template includes this image field:
Copy code
{
  "job_configuration": {
    "task_definition": {
      "containerDefinitions": [
        {
          "image": "{{ image }}",
          "name": "{{ container_name }}"
        }
      ],
      "cpu": "{{ cpu }}",
      "family": "{{ family }}",
      "memory": "{{ memory }}",
      "executionRoleArn": "{{ execution_role_arn }}"
    },
    "task_run_request": {
      "taskDefinition": "{{ task_definition_arn }}",
      "cluster": "{{ cluster }}",
      "overrides": {
        "containerOverrides": [
          {
            "name": "{{ container_name }}",
            "command": "{{ command }}",
            "environment": "{{ env }}",
            "cpu": "{{ cpu }}",
            "memory": "{{ memory }}"
          }
        ],
        "cpu": "{{ cpu }}",
        "memory": "{{ memory }}",
        "taskRoleArn": "{{ task_role_arn }}"
      }
    }
  }
}
Where to change the image: - If using a task definition ARN: update the image in the ECS task definition in AWS (create a new revision with your desired image). - If not using an ARN: edit the base job template for your ECS work pool and set the image (for example,
prefecthq/prefect:3.4.15-python3.10
). You can update the base job template in the Cloud UI (Work Pools > your ECS pool > Settings > Base job template) or with the CLI by exporting, editing, then updating:
Copy code
prefect work-pool inspect "<your-ecs-pool-name>" > pool.json
# edit pool.json -> set job_configuration.task_definition.containerDefinitions[0].image
prefect work-pool update "<your-ecs-pool-name>" --base-job-template pool.json
If you paste your YAML, I’ll point out the exact lines to change (image, task definition ARN, and any overrides).
t
@Marvin I tried pasting my yaml earlier but it was too large to be read by you, so I will send a small part of it. This is the PrefectTaskDefinition that is within the Resources: section of the yaml file.
Copy code
PrefectTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref PrefectFargateCluster
      Cpu: !Ref cpu
      Memory: !Ref memory
      NetworkMode: awsvpc
      ExecutionRoleArn: !Ref ExecutionRole
      TaskRoleArn: !Ref TaskRole
      ContainerDefinitions:
        - Name: !Sub ${environment}-prefect-container
          Image: !Ref image
          EntryPoint:
            - "bash"
            - "-c"
          StopTimeout: 120
          Environment:
            - Name: PREFECT_LOGGING_LEVEL
              Value: DEBUG
            - Name: AWS_RETRY_MODE
              Value: adaptive
            - Name: AWS_MAX_ATTEMPTS
              Value: 99
          Command:
            - !Sub "pip install prefect-aws && prefect worker start -t ecs -p ecs-worker-pool"
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref PrefectLogGroup
              awslogs-stream-prefix: prefect
          Secrets:
            - Name: PREFECT_API_URL
              ValueFrom: !FindInMap [ EnvConfig, !Ref environment, PrefectApiUrl ]
            - Name: PREFECT_API_KEY
              ValueFrom: !FindInMap [ EnvConfig, !Ref environment, PrefectApiKey ]
      RequiresCompatibilities:
        - FARGATE
Here is the Parameter defined for the image in the same file too:
Copy code
image:
    Type: String
    Description: Docker image from prefect which the worker will use
    Default: prefecthq/prefect:2.14.13-python3.10
m
thought for 99.3 seconds
Thanks for sharing the snippet — that helps a lot. What you’ve shown is the Task Definition for the long‑lived ECS task that runs your Prefect worker: - It uses the
image
parameter (defaulting to
prefecthq/prefect:2.14.13-python3.10
) - Then it runs this command at container start:
Copy code
pip install prefect-aws && prefect worker start -t ecs -p ecs-worker-pool
Why the UI shows Prefect 3.4.15: - The base image starts with Prefect 2.14.13. - At startup,
pip install prefect-aws
installs the latest
prefect-aws
, which depends on Prefect 3.x. Pip will upgrade the
prefect
package in-place to a 3.x version (e.g., 3.4.15). - The worker process then starts with Prefect 3.x, and the Cloud UI displays the worker’s running Prefect version — hence you see 3.4.15. You can confirm this in the worker’s logs (you should see pip upgrading
prefect
) or by running
prefect version
inside that container. How to make this consistent (pick one): 1) Move the worker to 3.x (recommended) - Keep your startup command but update the image parameter default to a 3.x tag:
Copy code
Default: prefecthq/prefect:3.4.15-python3.10
- Optionally pin the plugin to a 3.x-compatible range:
Copy code
pip install "prefect-aws>=0.5,<0.6" && prefect worker start -t ecs -p ecs-worker-pool
- Even better, bake
prefect-aws
into your image and remove the runtime pip install. 2) Keep the worker on 2.x (pin versions to prevent auto-upgrade) - Pin both
prefect
and
prefect-aws
in the startup command so pip doesn’t upgrade Prefect:
Copy code
pip install "prefect==2.14.13" "prefect-aws<0.5" && prefect worker start -t ecs -p ecs-worker-pool
- Or bake these exact versions into your image and remove the runtime pip install. Note on flow run images: - The snippet you shared is for the worker container only. The image used for actual flow runs is controlled by your ECS work pool’s base job template or a task definition ARN configured on the pool/deployment. You can inspect the pool template (which contains the
task_definition.containerDefinitions[0].image
field used for flow runs) with:
Copy code
prefect work-pool inspect "<your-ecs-pool-name>"
The default ECS template looks like:
Copy code
{
  "job_configuration": {
    "task_definition": {
      "containerDefinitions": [
        {
          "image": "{{ image }}",
          "name": "{{ container_name }}"
        }
      ],
      "cpu": "{{ cpu }}",
      "family": "{{ family }}",
      "memory": "{{ memory }}",
      "executionRoleArn": "{{ execution_role_arn }}"
    },
    "task_run_request": {
      "taskDefinition": "{{ task_definition_arn }}",
      "cluster": "{{ cluster }}",
      "overrides": {
        "containerOverrides": [
          {
            "name": "{{ container_name }}",
            "command": "{{ command }}",
            "environment": "{{ env }}",
            "cpu": "{{ cpu }}",
            "memory": "{{ memory }}"
          }
        ],
        "cpu": "{{ cpu }}",
        "memory": "{{ memory }}",
        "taskRoleArn": "{{ task_role_arn }}"
      }
    }
  }
}
- If you want your flow runs to use a specific Prefect version/image, set it there (or point to a specific task definition ARN). References: - Workers vs. work pools overview: Work pools and workers (blog) - Migrating/understanding workers in 3.x: Upgrade agents to workers - Managing deployments and images: prefect.yaml and deployments If you tell me whether you want to stick with 2.x or move to 3.x for the worker, I can suggest an updated CloudFormation snippet (image + command) and, if needed, show where to set the flow run image in your ECS work pool.