<@ULVA73B9P> Failed to submit flow run '64e4894b-2...
# ask-marvins
Saad80
09/24/2025, 3:07 AM@Marvin Failed to submit flow run '64e4894b-2c89-4727-be71-14e74afd1fc0' to infrastructure.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 1279, in _submit_run_and_capture_errors
await self._give_worker_labels_to_flow_run(flow_run.id)
File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 1562, in _give_worker_labels_to_flow_run
await self._client.update_flow_run_labels(flow_run_id, labels)
File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 1027, in update_flow_run_labels
response = await self.request(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration/base.py", line 53, in request
return await self._client.send(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 379, in send
response.raise_for_status()
File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 163, in raise_for_status
raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.cause
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '****/labels'
Response: {'detail': 'Not Found'}
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
080516 PM
prefect.flow_runs.worker
Reported flow run '64e4894b-2c89-4727-be71-14e74afd1fc0' as crashed: Flow run could not be submitted to infrastructure:
PrefectHTTPStatusError("Client error '404 Not Found' for url'*****/labels': {'detail': 'Not Found'}\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404")
m
Marvin
09/24/2025, 3:07 AM✅ thought for 241.2 seconds
Marvin
09/24/2025, 3:11 AMThanks for sharing the stack trace — that’s really helpful.
Short version: your worker is trying to update the flow run’s labels at the API route
/flow_runs/{id}/labelsprefect version<YOUR_PREFECT_API_URL>/docsPREFECT_API_ENABLE_LABELS/flow_runs/{id}/labelss
Saad80
09/24/2025, 3:15 AM@Marvin It seems that this is a prefect 3.xx thing but I'm supposed to be on prefect 2.14.20. My local machine is on 2.14.20 and I am self hosting.
m
Marvin
09/24/2025, 3:15 AM✅ thought for 63.0 seconds
Marvin
09/24/2025, 3:16 AMThanks — that context helps.
What your traceback shows is a Prefect 3.x client/worker trying to call the 3.x endpoint
/flow_runs/{id}/labels Copy code
prefect version Copy code
prefect version Copy code
pip install "prefect==2.14.20" Copy code
prefecthq/prefect:2.14.20-python3.11latestprefect versions
Saad80
09/24/2025, 3:28 AMon server hosting prefect:
prefect version
Version: 2.14.20
API version: 0.8.4
Python version: 3.9.23
Git commit: 8ceb0962
Built: Thu, Feb 1, 2024 6:30 PM
OS/Arch: linux/x86_64
Profile: ephemeral
Server type: server
and same locally @Marvin
m
Marvin
09/24/2025, 3:28 AM✅ thought for 43.8 seconds
Marvin
09/24/2025, 3:29 AMThanks for the details. Even though your server and your local CLI both report 2.14.20, the traceback shows a worker running in a different environment:
- It’s using Python 3.11 (
/usr/local/lib/python3.11/site-packages/.../flow_runs/{id}/labelsupdate_flow_run_labelsprefecthq/prefect:latest Copy code
which prefect
prefect version
python -c "import prefect, sys; print(prefect.__version__, sys.version)" Copy code
docker ps | grep prefect
docker exec -it <container_id> prefect version Copy code
kubectl get pods -n <ns>
kubectl exec -it <pod> -n <ns> -- prefect version Copy code
pip install "prefect==2.14.20" Copy code
prefecthq/prefect:2.14.20-python3.11:latestprefect versions
Saad80
09/24/2025, 4:37 PM@Marvin
16:36:27.305 | WARNING | prefect.events.clients - Unable to connect to 'ws://****.<http://ap-south-1.compute.amazonaws.com:4200/api/events/in|ap-south-1.compute.amazonaws.com:4200/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 403. Set PREFECT_DEBUG_MODE=1 to see the full error.m
Marvin
09/24/2025, 4:37 PM✅ thought for 116.9 seconds
Marvin
09/24/2025, 4:39 PMThat 403 on the events websocket is another strong signal that something in your runtime is using Prefect 3.x against your 2.14.20 server.
Why I say that:
- The log comes from
prefect.events.clients/api/events/in Copy code
which prefect
prefect version
python -c "import prefect, sys; print(prefect.__version__, sys.version)" Copy code
docker ps | grep prefect
docker exec -it <container_id> prefect version Copy code
kubectl get pods -n <ns>
kubectl exec -it <pod> -n <ns> -- prefect version:latestprefecthq/prefect:2.14.20-python3.11pip install "prefect==2.14.20" Copy code
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
location /api/ {
proxy_pass <http://prefect-server:4200;>
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_read_timeout 600s;
}/api/events/inPREFECT_API_URLhttps://.../apiwss://ws:// Copy code
export PREFECT_DEBUG_MODE=1/flow_runs/{id}/labelss
Saad80
09/24/2025, 5:30 PM@Marvin failure is at worker level but here is my task definition:
{
"compatibilities": [
"EC2",
"FARGATE"
],
"containerDefinitions": [
{
"command": [
"/bin/sh",
"-c",
"prefect version && python -c 'import pydantic, prefect; print(\"pydantic:\", pydantic.version)' && pip install --no-cache-dir 'prefect-aws==0.3.1' 'pydantic<2' 'boto3<2' && prefect worker start --pool ecs-two-flows --type ecs"
],
"cpu": 512,
"environment": [
{
"name": "PREFECT_API_URL",
"value": "<http://***:4200/api>"
}
],
"essential": true,
"image": "prefecthq/prefect:2.14.20-python3.11",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/prefect-worker-logs",
"awslogs-create-group": "true",
"awslogs-region": "ap-south-1",
"awslogs-stream-prefix": "ecs"
}
},
"memory": 1024,
"mountPoints": [],
"name": "prefect-worker",
"portMappings": [],
"systemControls": [],
"volumesFrom": []
}
],
"cpu": "512",
"executionRoleArn": "arnawsiam:971422704224role/two-flows-prefect-worker-role",
"family": "two-flows-prefect-worker-task",
"memory": "1024",
"networkMode": "awsvpc",
"placementConstraints": [],
"registeredAt": "2025-09-04T201609.300Z",
"registeredBy": "arnawsiam:971422704224root",
"requiresAttributes": [
{
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"name": "ecs.capability.execution-role-awslogs"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"name": "com.amazonaws.ecs.capability.task-iam-role"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"name": "ecs.capability.task-eni"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
}
],
"requiresCompatibilities": [
"FARGATE"
],
"revision": 6,
"status": "ACTIVE",
"taskDefinitionArn": "arnawsecsap south 1971422704224task definition/two flows prefect worker task6",
"taskRoleArn": "arnawsiam:971422704224role/two-flows-prefect-worker-role",
"volumes": [],
"tags": []
}
m
Marvin
09/24/2025, 5:30 PM✅ thought for 104.1 seconds
Marvin
09/24/2025, 5:31 PMGot it — thanks for sharing the ECS task definition. The root cause is in the container command:
- You start from an image pinned to Prefect 2.14.20 (good)
- Then you run pip install prefect-aws==0.3.1 right before starting the worker
prefect-aws 0.3.x is for Prefect 3.x. Installing it will upgrade Prefect inside the container from 2.14.20 to 3.x after your initial
prefect version/flow_runs/{id}/labelsprefect-aws<0.3.0 Copy code
/bin/sh -c "
pip install --no-cache-dir --upgrade --upgrade-strategy only-if-needed \
'prefect==2.14.20' 'prefect-aws<0.3.0' 'boto3<2' \
&& python -c 'import prefect, pydantic; print(\"prefect:\", prefect.__version__, \"pydantic:\", pydantic.__version__)' \
&& prefect version \
&& prefect worker start --pool ecs-two-flows --type ecs
"prefect-aws==0.3.x/api/events/inpython -c ...prefect version