Thanks for sharing the traceback — that’s very helpful. What’s happening: - Your ECS worker is trying to add its labels to the flow run via the endpoint

/api/flow_runs/{id}/labels

. - That endpoint returns 404 from your server, so the worker marks the flow run as crashed. Why this happens: - In Prefect 3.x, the server exposes the labels endpoint; Prefect 2.x does not. - A 404 here almost always means a version mismatch: a Prefect 3.x worker talking to a Prefect 2.x (or older 3.x) server that doesn’t have this route enabled yet. - There’s also a known report of this exact 404 when server and client were on different 3.x minors; aligning versions resolved it. See: Issue #16828 Docs for the endpoint (to confirm it should exist in 3.x): - Update flow run labels (REST) - Python client: flow run endpoints What I recommend: 1) Verify versions on both sides (worker and server) - On the worker host/container:

prefect version

This prints both the local Prefect version and the “API version” it’s connected to. - On the API server container/VM:

prefect version

Or check the UI footer for the server version. 2) Align the versions - If your server is on Prefect 2.x: upgrade your self-hosted server to Prefect 3.x (recommended to the latest 3.4.x). After upgrading, restart the server and apply any necessary DB migrations per the self-host docs. - If you’re already on 3.x, make sure the server and workers are on the same minor version. Upgrading both to the latest 3.4.x generally avoids these mismatches. Changelog for recent releases: 3.4.15 release notes 3) Quick test - From a machine that can reach your API, hit the route directly (replace the ID with a real one). A 404 here confirms the server doesn’t support the route yet:

curl -i http://<your-ec2-host>:4200/api/flow_runs/<flow_run_id>/labels

Temporary workaround (if you can’t upgrade the server immediately): - Pin your worker image/venv to a Prefect version that does not attempt label updates (e.g., users in the linked issue confirmed 3.0.0 worked for them):

pip install "prefect==3.0.0"

Or set your ECS worker’s image to a Prefect 3.0.0 tag. Then plan to upgrade both server and workers together later. Could you share: - The output of

prefect version

from your worker and your server - How you’re running the server (Docker Compose, ECS, VM install) - The Prefect image/tag you’re using for the ECS worker With that, I can give you an exact upgrade/alignment path.

Great question. A quick rundown: - Work pools don’t have a “version” themselves. What matters is: 1) the Prefect version running in your ECS worker container 2) the Prefect version of your API/server How to check the worker’s Prefect version (in ECS): - If ECS Exec is enabled on the task: 1) Open the ECS console → Cluster → Services → Tasks → select your worker task 2) Execute a command in the worker container:

prefect version

This will print something like:

Version:             3.x.y
     API version:         0.8.z
     ...

- If you can’t Exec, you can temporarily change the container command in the task definition to print the version before starting the worker:

sh -lc "prefect version && prefect worker start --pool 'YOUR_ECS_POOL_NAME'"

Then check the CloudWatch logs for the output. - Alternatively, if you can run Python in the container:

python -c "import prefect; print(prefect.__version__)"

- You can also check the container image tag in the task definition (Container → Image). If you’re using a Prefect image like

prefecthq/prefect:3.4.15-python3.12

, that indicates the bundled Prefect version. How to check the server/API version: - From any machine configured to talk to your server (PREFECT_API_URL pointing to your EC2 host:4200):

prefect version

This shows both your local client version and the “API version” reported by the server it’s connected to. - Or run the same command directly on the machine/container that’s running your Prefect server:

prefect version

- If you have the UI, check the footer; it usually shows the server version. If you share the outputs of

prefect version

from the worker container and the server, I can confirm if a version mismatch is causing your 404 and suggest the cleanest upgrade path.

You nailed it — the issue is the worker image. You’re running a 3.x worker (

prefecthq/prefect:3-latest

) against a 2.x server. In 3.x the worker calls

/api/flow_runs/{id}/labels

, which doesn’t exist in Prefect 2.x, causing that 404. Also, your

PREFECT_API_URL

in the task definition looks like it includes Slack/HTML formatting (

<...>

and

<...>

). That can corrupt the URL the client uses, which is why your error shows a duplicated bracketed URL. Make sure it’s a plain URL with no angle brackets or pipes. What to change to pin everything to 2.14.20: - Use a Prefect 2.14.20 image for the worker - Install a Prefect 2-compatible AWS integration - Fix the API URL value (no angle brackets) Example ECS task definition snippet:

"containerDefinitions": [
  {
    "name": "prefect-worker",
    "image": "prefecthq/prefect:2.14.20-python3.11",
    "cpu": 512,
    "memory": 1024,
    "essential": true,
    "command": [
      "/bin/sh",
      "-c",
      "pip install 'prefect-aws==0.3.*' && prefect worker start --pool ecs-two-flows --type ecs"
    ],
    "environment": [
      {
        "name": "PREFECT_API_URL",
        "value": "<http://ec2-13-233-32-245.ap-south-1.compute.amazonaws.com:4200/api>"
      }
      // If you use auth, add PREFECT_API_KEY here
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/prefect-worker-logs",
        "awslogs-region": "ap-south-1",
        "awslogs-stream-prefix": "ecs",
        "awslogs-create-group": "true"
      }
    }
  }
]

Notes: - The tag

2.14.20-python3.11

ensures you’re on Prefect 2.14.20 and a known Python runtime. If you prefer another Python minor, use the corresponding tag if available (e.g.,

2.14.20-python3.10

). You can also use

2.14.20

if that’s the only tag you see. - I’ve pinned

prefect-aws

to

0.3.*

, which is a safe series for Prefect 2.14.x. If you already have a known-good version in your environment, pin that exact version instead. After updating: 1) Redeploy the ECS service. 2) Exec into the worker task (or check logs) and verify:

prefect version

You should see: - Version: 2.14.20 - API version shown should match your server Reference (Prefect 2.x ECS worker guide): - https://docs-2.prefect.io/latest/integrations/prefect-aws/ecs_guide/ If you’d like, paste your updated task definition (with secrets redacted) and I’ll sanity-check it.

✅ thought for 46.9 seconds

Thanks for the update — those logs still show a 3.x worker. The giveaway is the call to

update_flow_run_labels

and the traceback path in `prefect/workers/base.py`; that’s a 3.x client method that doesn’t exist in Prefect 2.x. So your ECS task is almost certainly still running a 3.x image/config, which will 404 against a 2.x server. Do this to resolve: 1) Pin the worker image to Prefect 2.14.20 and pin prefect-aws - Update your task definition to something like:

"containerDefinitions": [
    {
      "name": "prefect-worker",
      "image": "prefecthq/prefect:2.14.20-python3.11",
      "cpu": 512,
      "memory": 1024,
      "essential": true,
      "command": [
        "/bin/sh",
        "-c",
        "prefect version && pip install 'prefect-aws==0.3.*' && prefect worker start --pool ecs-two-flows --type ecs"
      ],
      "environment": [
        {
          "name": "PREFECT_API_URL",
          "value": "<http://ec2-13-233-32-245.ap-south-1.compute.amazonaws.com:4200/api>"
        }
        // add PREFECT_API_KEY if your server is authenticated
      ]
    }
  ]

- Note the

prefect version &&

at the start — this prints the version in CloudWatch logs so you can verify the worker is really on 2.14.20. 2) Fix the API URL formatting - Ensure the value is plain text, no angle brackets or HTML entities. For example:

<http://ec2-13-233-32-245.ap-south-1.compute.amazonaws.com:4200/api>

not

<http://.../api>

3) Redeploy the ECS service to pick up the new task definition - Create a new task definition revision with the changes. - Update the ECS service to use that revision and “Force new deployment” so old tasks are replaced. - In the running task, confirm: - The image tag is

prefecthq/prefect:2.14.20-python3.11

- Logs show

Version: 2.14.20

4) Verify the server version - On the machine/container running your Prefect server, run:

prefect version

Confirm it’s also 2.14.20. If it’s not, align it. Reference - Prefect 2.x ECS worker guide: https://docs-2.prefect.io/latest/integrations/prefect-aws/ecs_guide/ - Prior report of this 404 when versions were misaligned: Issue #16828 If you want, paste the updated (redacted) task definition or the worker startup logs and I’ll sanity-check them.

Thanks for the snippet — I see two problems that explain why you’re still hitting the 404: 1) Your container is upgrading Prefect to 3.x at startup - You’re running

pip install prefect-aws

without pinning a 2.x-compatible version. The latest

prefect-aws

will pull Prefect 3.x as a dependency, upgrading the container and causing the worker to call the 3.x-only labels endpoint. - That’s why your traceback still shows

update_flow_run_labels

. 2) Your

PREFECT_API_URL

still has angle brackets - In your task definition it’s set to

"<http://REDACTED.../api>"

. Those angle brackets will be included literally and can corrupt the URL (you can see the mangled URL fragments in the error). It must be the plain URL with no

< >

. Fix - Pin the container image to Prefect 2.14.20 (you did this, good). - Pin prefect-aws to a Prefect 2-compatible release (0.3.x). - Remove angle brackets from the API URL. - Optional but recommended: print the Prefect version at startup so you can verify in CloudWatch logs. Example task definition snippet

"containerDefinitions": [
  {
    "name": "prefect-worker",
    "image": "prefecthq/prefect:2.14.20-python3.11",
    "cpu": 512,
    "memory": 1024,
    "essential": true,
    "command": [
      "/bin/sh",
      "-lc",
      "prefect version && pip install --no-cache-dir 'prefect-aws==0.3.10' 'boto3<2' && prefect worker start --pool ecs-two-flows --type ecs"
    ],
    "environment": [
      {
        "name": "PREFECT_API_URL",
        "value": "<http://REDACTED.ap-south-1.compute.amazonaws.com:4200/api>"
      }
      // add PREFECT_API_KEY if needed
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/prefect-worker-logs",
        "awslogs-region": "ap-south-1",
        "awslogs-stream-prefix": "ecs",
        "awslogs-create-group": "true"
      }
    }
  }
]

Notes - Make sure the command uses

&&

, not

&&

. Slack shows HTML entities, but your ECS JSON/console should have literal

&&

. -

prefect version

at startup lets you confirm in the logs that it’s truly 2.14.20. - Pinning

prefect-aws==0.3.10

(or any 0.3.x) prevents pip from upgrading Prefect to 3.x. - If you’d rather avoid installing at runtime, bake a small custom image FROM

prefecthq/prefect:2.14.20-python3.11

with

prefect-aws==0.3.10

preinstalled. After updating - Register a new task definition revision and “Force new deployment” on the ECS service so old tasks are replaced. - Check CloudWatch logs for: -

Version: 2.14.20

- No references to

update_flow_run_labels

in stack traces Docs - Prefect 2.x ECS worker guide: ECS worker guide (Prefect 2) If it still fails, please paste the first 10–15 lines of the worker’s startup logs (especially the

prefect version

output) and I’ll help you zero in on it.

Thanks — that error tells us two key things: - Pydantic v2 is active in your worker container (the message links to pydantic 2.x docs). - You installed a Prefect 2-era

prefect-aws

(0.3.x) that expects Pydantic v1, so the collection fails to load and the worker won’t start. Given you want everything on Prefect 2.14.20, let’s make sure your worker container is truly on Prefect 2 + Pydantic 1 and that we don’t accidentally upgrade it at runtime. What to change in your ECS task definition 1) Remove angle brackets and HTML entities - Your

PREFECT_API_URL

must be plain text, not

<...>

. - In the command, make sure it’s literal

&&

not

&&

(Slack shows entities; the ECS JSON should not). 2) Pin all versions explicitly and print them at startup - This guarantees you can see in CloudWatch logs what’s running and prevents pip from pulling Prefect 3 / Pydantic 2. Example container snippet

"containerDefinitions": [
  {
    "name": "prefect-worker",
    "image": "prefecthq/prefect:2.14.20-python3.11",
    "cpu": 512,
    "memory": 1024,
    "essential": true,
    "command": [
      "/bin/sh",
      "-lc",
      "prefect version && python -c 'import pydantic, prefect; print(\"pydantic:\", pydantic.__version__)' && pip install --no-cache-dir 'prefect-aws==0.3.10' 'pydantic<2' 'boto3<2' && prefect worker start --pool ecs-two-flows --type ecs"
    ],
    "environment": [
      {
        "name": "PREFECT_API_URL",
        "value": "<http://REDACTED.ap-south-1.compute.amazonaws.com:4200/api>"
      }
      // Add PREFECT_API_KEY if your server requires auth
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/prefect-worker-logs",
        "awslogs-region": "ap-south-1",
        "awslogs-stream-prefix": "ecs",
        "awslogs-create-group": "true"
      }
    }
  }
]

Important notes - The base image already contains Prefect 2.14.20 with Pydantic 1.x. Pinning

pydantic<2

is a safety net. - Pin

prefect-aws==0.3.10

(any 0.3.x is fine for 2.14.x). Do not install newer

prefect-aws

on Prefect 2 or it may attempt to drag in Prefect 3 / Pydantic 2. - If you prefer to avoid installing at runtime, build a tiny image:

FROM prefecthq/prefect:2.14.20-python3.11
  RUN pip install --no-cache-dir 'prefect-aws==0.3.10' 'pydantic<2' 'boto3<2'
  CMD ["prefect", "worker", "start", "--pool", "ecs-two-flows", "--type", "ecs"]

Then use that image in your task definition. After updating the task - Register a new task definition revision and Force new deployment on the ECS service so old tasks are replaced. - In CloudWatch logs, confirm: - Prefect prints

Version: 2.14.20

- The “pydantic:” line shows a 1.x version - No “Failed to load collection 'prefect_aws'” warnings - If you still see calls to

update_flow_run_labels

in tracebacks, that’s a 3.x client. Double-check the running task is using the updated image and command. Docs for reference - Prefect 2.x ECS worker guide: https://docs-2.prefect.io/latest/integrations/prefect-aws/ecs_guide If you can paste the first 10–15 lines of the worker startup logs (the

prefect version

and “pydantic:” lines), I’ll confirm the environment is correct.