Hi I ve been encountering below error with self hosted prefe Prefect Community #ask-community

Hi! I’ve been encountering below error with self-h...

Christine Chen

01/06/2025, 9:46 PM

Hi! I’ve been encountering below error with self-hosted prefect server. Have verified worker was deployed to the same namespace. Any ideas? Prefect version: 3.0.4 Server/worker chart version: 2024.10.1175730

Copy code

Failed to submit flow run '46744906-60cd-410c-a04c-12d7a80917cf' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 985, in _submit_run_and_capture_errors
    await self._give_worker_labels_to_flow_run(flow_run.id)
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 1232, in _give_worker_labels_to_flow_run
    await self._client.update_flow_run_labels(flow_run_id, labels)
  File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration/__init__.py", line 2950, in update_flow_run_labels
    response = await self._client.patch(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1933, in patch
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 355, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 163, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<http://prefect-server>.{namespace}.svc.cluster.local:4200/api/flow_runs/46744906-60cd-410c-a04c-12d7a80917cf/labels'
Response: {'detail': 'Not Found'}

✅ 1

Jake Kaplan

01/06/2025, 10:02 PM

hey, I believe that call to the

/labels

endpoint was only added in a more recent version. Maybe

3.1.9

? It seems like your worker may be on a newer prefect version than your server?

👀 1

Christine Chen

01/06/2025, 10:38 PM

Thanks! That was the issue. After pinning the image version to 3.0.4 in the values.yaml, the issue was resolved.

🙌 1

Praveen Mishra

01/15/2025, 3:23 PM

Hey @Jake Kaplan, I am on chart version: 2024.10.10155340 with imageTag on values.yaml as

3.1.4-python3.11

for server. getting this error. What am I missing here?

Jake Kaplan

01/15/2025, 3:27 PM

What version of prefect is your worker on? If it's the same as above, it's likely your worker is on a new version of prefect than your server.

Praveen Mishra

01/16/2025, 3:15 AM

k get pods -n dev-prefect prefect-worker-868c546b9d-v4g55 -o yaml

gives: prefect-version: 2.19.8

Jake Kaplan

01/16/2025, 3:21 AM

Can you share the error you're getting? It's not possible for the traceback to be exactly the same as the above given your version. Your worker and server should also share a version. v2 clients are not fully compatible with a v3 server (and vice versa)

Praveen Mishra

01/16/2025, 3:47 AM

Copy code

Jan 15th, 2025
Worker 'KubernetesWorker c6790ecd-b960-4c00-a33f-fb874bcd8035' submitting flow run '0fefc299-c8d3-4a59-8355-fffc4d41d91b'
07:45:28 PM
prefect.flow_runs.worker
Failed to submit flow run '0fefc299-c8d3-4a59-8355-fffc4d41d91b' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/workers/base.py", line 987, in _submit_run_and_capture_errors
    await self._give_worker_labels_to_flow_run(flow_run.id)
  File "/usr/local/lib/python3.12/site-packages/prefect/workers/base.py", line 1235, in _give_worker_labels_to_flow_run
    await self._client.update_flow_run_labels(flow_run_id, labels)
  File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 897, in update_flow_run_labels
    response = await self.request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/base.py", line 46, in request
    return await self._client.request(method, path, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 355, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 163, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<http://prefect-server.dev-prefect.svc.cluster.local:4200/api/flow_runs/0fefc299-c8d3-4a59-8355-fffc4d41d91b/labels>'
Response: {'detail': 'Not Found'}
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404>
07:45:28 PM
prefect.flow_runs.worker
Reported flow run '0fefc299-c8d3-4a59-8355-fffc4d41d91b' as crashed: Flow run could not be submitted to infrastructure:
PrefectHTTPStatusError("Client error '404 Not Found' for url '<http://prefect-server.dev-prefect.svc.cluster.local:4200/api/flow_runs/0fefc299-c8d3-4a59-8355-fffc4d41d91b/labels>'\nResponse: {'detail': 'Not Found'}\nFor more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404>")

Jake Kaplan

01/16/2025, 3:54 AM

It seems like your worker running a later version than the yaml is saying?

_give_worker_labels_to_flow_run

was not added to

2.x

If that's the case it's the same issue as above, where your worker is on a later version than your server. I would recommend exec-ing to both your worker and server pod and running

prefect version

to get the versions, and then making sure they match up.

Praveen Mishra

01/16/2025, 4:09 AM

worker:

Defaulted container "prefect-worker" out of: prefect-worker, sync-base-job-template (init)

$ prefect version Version: 3.1.12 API version: 0.8.4 Python version: 3.12.8 Git commit: e299e5a7 Built: Thu, Jan 9, 2025 10:09 AM OS/Arch: linux/x86_64 Profile: ephemeral Server type: server Pydantic version: 2.10.5 Integrations: prefect-kubernetes: 0.5.3

Praveen Mishra

01/16/2025, 4:10 AM

server:

Copy code

$ prefect version
04:10:22.204 | DEBUG   | prefect.profiles - Using profile 'ephemeral'
Version:             3.1.4
API version:         0.8.4
Python version:      3.11.10
Git commit:          78ee41cb
Built:               Wed, Nov 20, 2024 7:37 PM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         server
Pydantic version:    2.10.0

Jake Kaplan

01/16/2025, 4:11 AM

So your server is a few versions behind your worker it seems. • server is on

3.1.4

• worker is on

3.1.12

The endpoint the worker is looking for was only added in

3.1.9

. You'll need to either downgrade your worker, or upgrade your server so that the versions matches

👀 1

Praveen Mishra

01/16/2025, 1:27 PM

that fixed it. thanks much!

🙌 1

98 Views

Open in Slack

Previous Next