Hi all; I am running prefect on an on prem kubernetes cluster with ingress enabled and set as the api-url and the prefect agent pointing at the ingress api-url. After upgrading to prefect 2.10.21 my agent got the following error when attempting to start:

prefect.exceptions.PrefectHTTPStatusError: Client error '422 Unprocessable Entity' for url '<https://my-prefect.com/api/flow_runs/filter>'
Response: {'exception_message': 'Invalid request received.', exception_detail': [{'loc': ['path', 'id'], 'msg': value is not a valid uuid', 'type': 'type_error.uuid'}], 'request_body': None}

Has anyone else seen this issue? If so, what did they do to resolve it? This seems similar to other issues people have had related to running different versions on server vs client. However I have already verified that my prefect server and client are both running 2.10.21.

hey, could there have been a change in your ingress setup? that error to me looks like it's not making a post request. for example:

import asyncio
from prefect.client.orchestration import get_client

async def main():
    async with get_client() as client:
        response = await <http://client._client.post|client._client.post>("/flows/filter")
        print(f"got status code: {response.status_code}")
        print(f"found {len(response.json())} flow runs")
        await client._client.get("/flows/filter")

if __name__ == '__main__':
    asyncio.run(main())

gives

200
7
Traceback (most recent call last):
  File "/Users/jakekaplan/PycharmProjects/demo-flows/test_client.py", line 15, in <module>
    asyncio.run(main())
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/jakekaplan/PycharmProjects/demo-flows/test_client.py", line 10, in main
    response = await client._client.get("/flows/filter")
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/site-packages/prefect/client/base.py", line 280, in send
    response.raise_for_status()
  File "/Users/jakekaplan/opt/anaconda3/envs/demo-flows/lib/python3.10/site-packages/prefect/client/base.py", line 138, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '422 Unprocessable Entity' for url '<http://ephemeral-prefect/api/flows/filter>'
Response: {'exception_message': 'Invalid request received.', 'exception_detail': [{'loc': ['path', 'id'], 'msg': 'value is not a valid uuid', 'type': 'type_error.uuid'}], 'request_body': None}

@Jake Kaplan Thanks for this response. A change in the ingress is a possibility -- let me check that and get back to you.

Sorry for the delayed response here but glad you were able to figure it out!