Hi all, I am using Prefect 2.14.15 and have my own prefect Orion server with multiple work-pools deployed on different servers. Flows running on my machine is failing due to PrefectHTTPStatusError for some of the internal prefect api like /flow_run, /flows, /set_state. Error Traceback:

Encountered exception during execution:
Traceback (most recent call last):
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/engine.py", line 849, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 293, in aresult
    return await asyncio.wrap_future(self.future)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/var/www/apt/apt/prefect_flows/workflow.py", line 20, in execute_workflow
    return state.result()
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/client/schemas/objects.py", line 212, in result
    return get_state_result(self, raise_on_failure=raise_on_failure, fetch=fetch)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/states.py", line 71, in get_state_result
    return _get_state_result(state, raise_on_failure=raise_on_failure)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
    return call()
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 398, in _call_
    return self.result()
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 284, in result
    return self.future.result(timeout=timeout)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 168, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 355, in _run_async
    result = await coro
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/states.py", line 91, in _get_state_result
    raise await get_state_exception(state)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/engine.py", line 849, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 293, in aresult
    return await asyncio.wrap_future(self.future)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/var/www/apt/apt/prefect_flows/workflow.py", line 28, in run_node_flow
    res = run_workflow(workflow, tenant, payload_data)
  File "/var/www/apt/apt/prefect_flows/workflow.py", line 48, in run_workflow
    process_conditional_task(task, tenant, result_of_tasks)
  File "/var/www/apt/apt/prefect_flows/workflow.py", line 95, in process_conditional_task
    filtered_data = check_condition(task.get("condition"), data_to_filter)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/tasks.py", line 569, in _call_
    return enter_task_run_engine(
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/engine.py", line 1382, in enter_task_run_engine
    return from_sync.wait_for_call_in_loop_thread(begin_run)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/api.py", line 243, in wait_for_call_in_loop_thread
    return call.result()
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 284, in result
    return self.future.result(timeout=timeout)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 168, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 355, in _run_async
    result = await coro
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/engine.py", line 1550, in get_task_call_return_value
    return await future._result()
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/futures.py", line 237, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/states.py", line 91, in _get_state_result
    raise await get_state_exception(state)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/task_runners.py", line 231, in submit
    result = await call()
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/engine.py", line 1780, in begin_task_run
    state = await orchestrate_task_run(
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/engine.py", line 2023, in orchestrate_task_run
    task_run = await client.read_task_run(task_run.id)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/client/orchestration.py", line 2031, in read_task_run
    response = await self._client.get(f"/task_runs/{task_run_id}")
  File "/home/azureuser/.local/lib/python3.10/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
  File "/home/azureuser/.local/lib/python3.10/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/client/base.py", line 312, in send
    response.raise_for_status()
  File "/home/azureuser/.local/lib/python3.10/site-packages/prefect/client/base.py", line 164, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc._cause_
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://prefect-server/api/task_runs/><task_id>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500>

Similarly for other APIs also,

prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<server>/api/task_runs/'
Response: {'exception_message': 'Internal Server Error'}

prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<server>/api/task_runs/<taskid>/set_state'
Response: {'exception_message': 'Internal Server Error'}
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<server>/api/flows/'
Response: {'exception_message': 'Internal Server Error'}

I am not able to figure out the root cause of this issue. My infra setup: - prefect-orion server: 16 GB RAM, 4 Core work-pools: - default-pool: same as prefect-orion server | 1 agent - pool1: server2 16 GB RAM, 4 core | 5 agents - pool2: same server as pool1 | 7 agents Pool1 has daily load of nearly 30k flows out of which ~5% failed due to these error. Pool2 has daily load of nearly 10k flows out of which ~8% is failed due to these error. Currently there are around 350k total flow runs until know. All the machines are on same VPC. I am using postgresQL DB connected to my prefect orion server with some table recods (like log, flow_run_state) nearly reaching 700k entries. Earlier i thought it was due to huge number of data in DB with is causing timeout but i had removed nearly 30% on my old data from DB but the error still exists. Does anyone have ony idea what might be the cause of it?