I have a prefect flow (k8s job) spinning up parallel subflows on the same pod and collecting results using

asynicio.gather

. I keep getting the following error on some of the runs without really seeing a meaningful pattern. Wondering if anybodyhas an idea.

07:17:37.254 | INFO    | Flow run 'inference-ba8422f7-9724-4708-bc6e-8e9b2623cba4' - Created subflow run 'topaz-squirrel' for flow 'perform-inference-subflow'
07:17:37.255 | INFO    | Flow run 'topaz-squirrel' - View at <http://prefect-server.prefect.svc.cluster.local:4200/flow-runs/flow-run/3de950a8-5a56-4988-b2b0-48525d47014f>
07:17:43.786 | ERROR   | Flow run 'inference-ba8422f7-9724-4708-bc6e-8e9b2623cba4' - Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 841, in orchestrate_flow_run
    result = await flow_call.aresult()
File "/usr/local/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 291, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
    result = await coro
  File "/opt/prefect/orchestrator-batch-inference/flows/perform_inference.py", line 122, in perform_inference
    seldon_responses = await asyncio.gather(*parallel_subflows)
  File "/usr/local/lib/python3.8/site-packages/prefect/_internal/concurrency/api.py", line 182, in wait_for_call_in_loop_thread
    return call.result()
  File "/usr/local/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 282, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 168, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
    result = await coro
  File "/usr/local/lib/python3.8/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 633, in create_and_begin_subflow_run
    flow_run = await client.create_flow_run(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/orchestration.py", line 609, in create_flow_run
    response = await <http://self._client.post|self._client.post>("/flow_runs/", json=flow_run_create_json)
File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 312, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 164, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://prefect-server.prefect.svc.cluster.local:4200/api/flow_runs/>'
Response: {'exception_message': 'Internal Server Error'}