Hi all; I am running prefect on an on prem kuberne...
# prefect-community
b
Hi all; I am running prefect on an on prem kubernetes cluster with ingress enabled and set as the api-url and the prefect agent pointing at the ingress api-url. After upgrading to prefect 2.10.21 my agent got the following error when attempting to start:
Copy code
prefect.exceptions.PrefectHTTPStatusError: Client error '422 Unprocessable Entity' for url '<https://my-prefect.com/api/flow_runs/filter>'
Response: {'exception_message': 'Invalid request received.', exception_detail': [{'loc': ['path', 'id'], 'msg': value is not a valid uuid', 'type': 'type_error.uuid'}], 'request_body': None}
Has anyone else seen this issue? If so, what did they do to resolve it? This seems similar to other issues people have had related to running different versions on server vs client. However I have already verified that my prefect server and client are both running 2.10.21.
1
j
hey, could there have been a change in your ingress setup? that error to me looks like it's not making a post request. for example:
Copy code
import asyncio
from prefect.client.orchestration import get_client

async def main():
    async with get_client() as client:
        response = await <http://client._client.post|client._client.post>("/flows/filter")
        print(f"got status code: {response.status_code}")
        print(f"found {len(response.json())} flow runs")
        await client._client.get("/flows/filter")

if __name__ == '__main__':
    asyncio.run(main())
gives
Copy code
200
7
Traceback (most recent call last):
  File "/Users/jakekaplan/PycharmProjects/demo-flows/test_client.py", line 15, in <module>
    asyncio.run(main())
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/jakekaplan/PycharmProjects/demo-flows/test_client.py", line 10, in main
    response = await client._client.get("/flows/filter")
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/site-packages/prefect/client/base.py", line 280, in send
    response.raise_for_status()
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/site-packages/prefect/client/base.py", line 138, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '422 Unprocessable Entity' for url '<http://ephemeral-prefect/api/flows/filter>'
Response: {'exception_message': 'Invalid request received.', 'exception_detail': [{'loc': ['path', 'id'], 'msg': 'value is not a valid uuid', 'type': 'type_error.uuid'}], 'request_body': None}
b
@Jake Kaplan Thanks for this response. A change in the ingress is a possibility -- let me check that and get back to you.
@Jake Kaplan I just double checked my ingress and re-pulled the most recent helm charts to be sure that they matched the most recent release exactly, and I still got the same error. I also reset my whole prefect service so I was starting from scratch (only default-agent-pool, nothing else set up) and tried to deploy an agent to it and still got that error. Do you mind sharing what your ingress looks like? I can do a comparison to see if mine is missing something. Otherwise could it be something with the source code?
@Jake Kaplan Never mind, I identified the issue. It took a prayer or two 😅 but I was able to figure out that the cause of the error was in fact related to ingress like you mentioned, but it was because in my values.yaml file I was setting the api-url manually instead of using the 'publicApiUrl' entry in the file. I had previously not used that because of a bug in a prior version, but using that appears to solve the issues. Thanks for your help Jake 🙂
🙌 1
j
Sorry for the delayed response here but glad you were able to figure it out!