Thomas DeBroux
09/10/2025, 1:59 PMprefecthq/prefect:2.14.13-python3.10Marvin
09/10/2025, 1:59 PMMarvin
09/10/2025, 2:06 PMprefecthq/prefect:2.14.13-python3.10). That image’s Prefect version is independent from the worker’s.
This mismatch can happen (and can work), because:
- The worker’s job is to poll the work pool and provision/run infrastructure (ECS tasks) according to your pool/template/deployment settings.
- The flow actually executes inside the container image you specify for the ECS task. That image can have a different Prefect version than the worker.
That said, for consistency and access to 3.x features, we generally recommend aligning the runtime image with 3.x as well (for example, prefecthq/prefect:3.4.15-python3.10 or a 3.x tag), and using the 3.x deployment patterns. If you stick with a 2.x image, some 3.x features and patterns (e.g., deployment APIs) won’t be available inside that container.
Helpful references:
- Workers and work pools overview (what the worker does vs. where the flow runs): Blog: Work pools and workers and Upgrade agents to workers (3.x)
- Managing deployments (where you can control the runtime image via your pool/job template or deployment config): prefect.yaml and deployments
If you want, share how you’re configuring the ECS work pool (base job template or task definition ARN) and I can point you to the exact field to change to upgrade the runtime image.Marvin
09/10/2025, 2:17 PMThomas DeBroux
09/10/2025, 2:19 PMMarvin
09/10/2025, 2:19 PMMarvin
09/10/2025, 2:24 PMprefect work-pool inspect "<your-ecs-pool-name>"
For reference, the default ECS base job template includes this image field:
{
"job_configuration": {
"task_definition": {
"containerDefinitions": [
{
"image": "{{ image }}",
"name": "{{ container_name }}"
}
],
"cpu": "{{ cpu }}",
"family": "{{ family }}",
"memory": "{{ memory }}",
"executionRoleArn": "{{ execution_role_arn }}"
},
"task_run_request": {
"taskDefinition": "{{ task_definition_arn }}",
"cluster": "{{ cluster }}",
"overrides": {
"containerOverrides": [
{
"name": "{{ container_name }}",
"command": "{{ command }}",
"environment": "{{ env }}",
"cpu": "{{ cpu }}",
"memory": "{{ memory }}"
}
],
"cpu": "{{ cpu }}",
"memory": "{{ memory }}",
"taskRoleArn": "{{ task_role_arn }}"
}
}
}
}
Where to change the image:
- If using a task definition ARN: update the image in the ECS task definition in AWS (create a new revision with your desired image).
- If not using an ARN: edit the base job template for your ECS work pool and set the image (for example, prefecthq/prefect:3.4.15-python3.10).
You can update the base job template in the Cloud UI (Work Pools > your ECS pool > Settings > Base job template) or with the CLI by exporting, editing, then updating:
prefect work-pool inspect "<your-ecs-pool-name>" > pool.json
# edit pool.json -> set job_configuration.task_definition.containerDefinitions[0].image
prefect work-pool update "<your-ecs-pool-name>" --base-job-template pool.json
If you paste your YAML, I’ll point out the exact lines to change (image, task definition ARN, and any overrides).Thomas DeBroux
09/10/2025, 2:35 PMPrefectTaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: !Ref PrefectFargateCluster
Cpu: !Ref cpu
Memory: !Ref memory
NetworkMode: awsvpc
ExecutionRoleArn: !Ref ExecutionRole
TaskRoleArn: !Ref TaskRole
ContainerDefinitions:
- Name: !Sub ${environment}-prefect-container
Image: !Ref image
EntryPoint:
- "bash"
- "-c"
StopTimeout: 120
Environment:
- Name: PREFECT_LOGGING_LEVEL
Value: DEBUG
- Name: AWS_RETRY_MODE
Value: adaptive
- Name: AWS_MAX_ATTEMPTS
Value: 99
Command:
- !Sub "pip install prefect-aws && prefect worker start -t ecs -p ecs-worker-pool"
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-region: !Ref AWS::Region
awslogs-group: !Ref PrefectLogGroup
awslogs-stream-prefix: prefect
Secrets:
- Name: PREFECT_API_URL
ValueFrom: !FindInMap [ EnvConfig, !Ref environment, PrefectApiUrl ]
- Name: PREFECT_API_KEY
ValueFrom: !FindInMap [ EnvConfig, !Ref environment, PrefectApiKey ]
RequiresCompatibilities:
- FARGATE
Here is the Parameter defined for the image in the same file too:
image:
Type: String
Description: Docker image from prefect which the worker will use
Default: prefecthq/prefect:2.14.13-python3.10Marvin
09/10/2025, 2:35 PMMarvin
09/10/2025, 2:37 PMimage parameter (defaulting to prefecthq/prefect:2.14.13-python3.10)
- Then it runs this command at container start:
pip install prefect-aws && prefect worker start -t ecs -p ecs-worker-pool
Why the UI shows Prefect 3.4.15:
- The base image starts with Prefect 2.14.13.
- At startup, pip install prefect-aws installs the latest prefect-aws, which depends on Prefect 3.x. Pip will upgrade the prefect package in-place to a 3.x version (e.g., 3.4.15).
- The worker process then starts with Prefect 3.x, and the Cloud UI displays the worker’s running Prefect version — hence you see 3.4.15.
You can confirm this in the worker’s logs (you should see pip upgrading prefect) or by running prefect version inside that container.
How to make this consistent (pick one):
1) Move the worker to 3.x (recommended)
- Keep your startup command but update the image parameter default to a 3.x tag:
Default: prefecthq/prefect:3.4.15-python3.10
- Optionally pin the plugin to a 3.x-compatible range:
pip install "prefect-aws>=0.5,<0.6" && prefect worker start -t ecs -p ecs-worker-pool
- Even better, bake prefect-aws into your image and remove the runtime pip install.
2) Keep the worker on 2.x (pin versions to prevent auto-upgrade)
- Pin both prefect and prefect-aws in the startup command so pip doesn’t upgrade Prefect:
pip install "prefect==2.14.13" "prefect-aws<0.5" && prefect worker start -t ecs -p ecs-worker-pool
- Or bake these exact versions into your image and remove the runtime pip install.
Note on flow run images:
- The snippet you shared is for the worker container only. The image used for actual flow runs is controlled by your ECS work pool’s base job template or a task definition ARN configured on the pool/deployment. You can inspect the pool template (which contains the task_definition.containerDefinitions[0].image field used for flow runs) with:
prefect work-pool inspect "<your-ecs-pool-name>"
The default ECS template looks like:
{
"job_configuration": {
"task_definition": {
"containerDefinitions": [
{
"image": "{{ image }}",
"name": "{{ container_name }}"
}
],
"cpu": "{{ cpu }}",
"family": "{{ family }}",
"memory": "{{ memory }}",
"executionRoleArn": "{{ execution_role_arn }}"
},
"task_run_request": {
"taskDefinition": "{{ task_definition_arn }}",
"cluster": "{{ cluster }}",
"overrides": {
"containerOverrides": [
{
"name": "{{ container_name }}",
"command": "{{ command }}",
"environment": "{{ env }}",
"cpu": "{{ cpu }}",
"memory": "{{ memory }}"
}
],
"cpu": "{{ cpu }}",
"memory": "{{ memory }}",
"taskRoleArn": "{{ task_role_arn }}"
}
}
}
}
- If you want your flow runs to use a specific Prefect version/image, set it there (or point to a specific task definition ARN).
References:
- Workers vs. work pools overview: Work pools and workers (blog)
- Migrating/understanding workers in 3.x: Upgrade agents to workers
- Managing deployments and images: prefect.yaml and deployments
If you tell me whether you want to stick with 2.x or move to 3.x for the worker, I can suggest an updated CloudFormation snippet (image + command) and, if needed, show where to set the flow run image in your ECS work pool.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by