https://prefect.io
Join Slack
Hi! I’ve been encountering below error with self-hosted prefect server. Have verified worker was dep...
# ask-community
c
Hi! I’ve been encountering below error with self-hosted prefect server. Have verified worker was deployed to the same namespace. Any ideas? Prefect version: 3.0.4 Server/worker chart version: 2024.10.1175730
Copy code
Failed to submit flow run '46744906-60cd-410c-a04c-12d7a80917cf' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 985, in _submit_run_and_capture_errors
    await self._give_worker_labels_to_flow_run(flow_run.id)
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 1232, in _give_worker_labels_to_flow_run
    await self._client.update_flow_run_labels(flow_run_id, labels)
  File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration/__init__.py", line 2950, in update_flow_run_labels
    response = await self._client.patch(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1933, in patch
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 355, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 163, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<http://prefect-server>.{namespace}.svc.cluster.local:4200/api/flow_runs/46744906-60cd-410c-a04c-12d7a80917cf/labels'
Response: {'detail': 'Not Found'}
✅ 1
j
hey, I believe that call to the 
/labels
endpoint was only added in a more recent version. Maybe 
3.1.9
? It seems like your worker may be on a newer prefect version than your server?
👀 1
c
Thanks! That was the issue. After pinning the image version to 3.0.4 in the values.yaml, the issue was resolved.
🙌 1
p
Hey @Jake Kaplan, I am on chart version: 2024.10.10155340 with imageTag on values.yaml as 
3.1.4-python3.11
for server. getting this error. What am I missing here?
j
What version of prefect is your worker on? If it's the same as above, it's likely your worker is on a new version of prefect than your server.
p
k get pods -n dev-prefect prefect-worker-868c546b9d-v4g55 -o yaml
gives: prefect-version: 2.19.8
j
Can you share the error you're getting? It's not possible for the traceback to be exactly the same as the above given your version. Your worker and server should also share a version. v2 clients are not fully compatible with a v3 server (and vice versa)
p
Copy code
Jan 15th, 2025
Worker 'KubernetesWorker c6790ecd-b960-4c00-a33f-fb874bcd8035' submitting flow run '0fefc299-c8d3-4a59-8355-fffc4d41d91b'
07:45:28 PM
prefect.flow_runs.worker
Failed to submit flow run '0fefc299-c8d3-4a59-8355-fffc4d41d91b' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/workers/base.py", line 987, in _submit_run_and_capture_errors
    await self._give_worker_labels_to_flow_run(flow_run.id)
  File "/usr/local/lib/python3.12/site-packages/prefect/workers/base.py", line 1235, in _give_worker_labels_to_flow_run
    await self._client.update_flow_run_labels(flow_run_id, labels)
  File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 897, in update_flow_run_labels
    response = await self.request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/base.py", line 46, in request
    return await self._client.request(method, path, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 355, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 163, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<http://prefect-server.dev-prefect.svc.cluster.local:4200/api/flow_runs/0fefc299-c8d3-4a59-8355-fffc4d41d91b/labels>'
Response: {'detail': 'Not Found'}
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404>
07:45:28 PM
prefect.flow_runs.worker
Reported flow run '0fefc299-c8d3-4a59-8355-fffc4d41d91b' as crashed: Flow run could not be submitted to infrastructure:
PrefectHTTPStatusError("Client error '404 Not Found' for url '<http://prefect-server.dev-prefect.svc.cluster.local:4200/api/flow_runs/0fefc299-c8d3-4a59-8355-fffc4d41d91b/labels>'\nResponse: {'detail': 'Not Found'}\nFor more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404>")
j
It seems like your worker running a later version than the yaml is saying? 
_give_worker_labels_to_flow_run
was not added to 
2.x
If that's the case it's the same issue as above, where your worker is on a later version than your server. I would recommend exec-ing to both your worker and server pod and running 
prefect version
to get the versions, and then making sure they match up.
p
worker: 
Defaulted container "prefect-worker" out of: prefect-worker, sync-base-job-template (init)
$ prefect version Version: 3.1.12 API version: 0.8.4 Python version: 3.12.8 Git commit: e299e5a7 Built: Thu, Jan 9, 2025 10:09 AM OS/Arch: linux/x86_64 Profile: ephemeral Server type: server Pydantic version: 2.10.5 Integrations: prefect-kubernetes: 0.5.3
server:
Copy code
$ prefect version
04:10:22.204 | DEBUG   | prefect.profiles - Using profile 'ephemeral'
Version:             3.1.4
API version:         0.8.4
Python version:      3.11.10
Git commit:          78ee41cb
Built:               Wed, Nov 20, 2024 7:37 PM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         server
Pydantic version:    2.10.0
j
So your server is a few versions behind your worker it seems. • server is on 
3.1.4
• worker is on 
3.1.12
The endpoint the worker is looking for was only added in 
3.1.9
. You'll need to either downgrade your worker, or upgrade your server so that the versions matches
👀 1
p
that fixed it. thanks much!
🙌 1
274 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by