I am seeing these errors in my actual workflows as...
# ask-community
t
I am seeing these errors in my actual workflows as well:
Copy code
Encountered exception during execution:
Traceback (most recent call last):
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/engine.py", line 833, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 292, in aresult
    return await asyncio.wrap_future(self.future)
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 316, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/scratch3/gal16b/packages/flint/flint/prefect/flows/continuum_pipeline.py", line 457, in process_bandpass_science_fields
    field_science_mss = task_flatten_prefect_futures(science_fields)
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/tasks.py", line 533, in __call__
    return enter_task_run_engine(
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/engine.py", line 1137, in enter_task_run_engine
    return from_sync.wait_for_call_in_loop_thread(begin_run)
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/api.py", line 243, in wait_for_call_in_loop_thread
    return call.result()
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
    return self.future.result(timeout=timeout)
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
    return self.__get_result()
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
    result = await coro
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/engine.py", line 1302, in get_task_call_return_value
    return await future._result()
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/futures.py", line 237, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
  File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/states.py", line 84, in _get_state_result
    raise UnfinishedRun(
prefect.exceptions.UnfinishedRun: Run is in PENDING state, its result is not available.
These particular flows (same code and same data) work perfect in isolation (e.g. when I run them by themselves without any other workflows running). Given I know I am seeing issues with load on the server/database, I am thinking this particular error is related and maybe some type of race condition? Is it a case that calls through the API that would interact with the postgres database are getting out of order across multiple uvicorn workers? I am also seeing messages like this in my prefect server logs
Copy code
Invalid HTTP request received.
Invalid HTTP request received.
Invalid HTTP request received.