Tim Galvin
11/09/2023, 4:21 AMEncountered exception during execution:
Traceback (most recent call last):
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/engine.py", line 833, in orchestrate_flow_run
result = await flow_call.aresult()
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 292, in aresult
return await asyncio.wrap_future(self.future)
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 316, in _run_sync
result = self.fn(*self.args, **self.kwargs)
File "/scratch3/gal16b/packages/flint/flint/prefect/flows/continuum_pipeline.py", line 457, in process_bandpass_science_fields
field_science_mss = task_flatten_prefect_futures(science_fields)
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/tasks.py", line 533, in __call__
return enter_task_run_engine(
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/engine.py", line 1137, in enter_task_run_engine
return from_sync.wait_for_call_in_loop_thread(begin_run)
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/api.py", line 243, in wait_for_call_in_loop_thread
return call.result()
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
return self.future.result(timeout=timeout)
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
return self.__get_result()
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
result = await coro
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/engine.py", line 1302, in get_task_call_return_value
return await future._result()
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/futures.py", line 237, in _result
return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
File "/scratch3/gal16b/mambaforge/envs/flint/lib/python3.8/site-packages/prefect/states.py", line 84, in _get_state_result
raise UnfinishedRun(
prefect.exceptions.UnfinishedRun: Run is in PENDING state, its result is not available.
These particular flows (same code and same data) work perfect in isolation (e.g. when I run them by themselves without any other workflows running). Given I know I am seeing issues with load on the server/database, I am thinking this particular error is related and maybe some type of race condition? Is it a case that calls through the API that would interact with the postgres database are getting out of order across multiple uvicorn workers?
I am also seeing messages like this in my prefect server logs
Invalid HTTP request received.
Invalid HTTP request received.
Invalid HTTP request received.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by