Ben Ayers-Glassey
04/21/2023, 5:50 PM2023-04-21T15:16:04.920982508Z /usr/local/lib/python3.8/runpy.py:127: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; thi
s may result in unpredictable behaviour
2023-04-21T15:16:04.921075059Z warn(RuntimeWarning(msg))
2023-04-21T15:16:04.997977913Z 15:16:04.996 | INFO | Flow run 'phi321-lysenko-tensor' - Created task run 'generate_operation_id-a9e17ed9-0' for task 'generate_operation_id'
2023-04-21T15:16:04.999103995Z 15:16:04.997 | INFO | Flow run 'phi321-lysenko-tensor' - Executing 'generate_operation_id-a9e17ed9-0' immediately...
2023-04-21T15:16:05.043778806Z 15:16:05.042 | INFO | Task run 'generate_operation_id-a9e17ed9-0' - Task run '43baff18-8292-4cd7-a694-2dcf6d1a0c77' already finished.
2023-04-21T15:16:05.119806977Z 15:16:05.044 | ERROR | Flow run 'phi321-lysenko-tensor' - Encountered exception during execution:
2023-04-21T15:16:05.119866174Z Traceback (most recent call last):
2023-04-21T15:16:05.119875789Z File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 637, in orchestrate_flow_run
2023-04-21T15:16:05.119882275Z result = await run_sync(flow_call)
2023-04-21T15:16:05.119888470Z File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
2023-04-21T15:16:05.119896590Z return await anyio.to_thread.run_sync(
2023-04-21T15:16:05.119904639Z File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
2023-04-21T15:16:05.119911921Z return await get_asynclib().run_sync_in_worker_thread(
2023-04-21T15:16:05.119917612Z File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
2023-04-21T15:16:05.119924062Z return await future
...etc...
2023-04-21T15:16:05.120058513Z File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
2023-04-21T15:16:05.120063946Z raise self._exception
2023-04-21T15:16:05.120069588Z File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 1068, in get_task_call_return_value
2023-04-21T15:16:05.120078014Z return await future._result()
2023-04-21T15:16:05.120085086Z File "/usr/local/lib/python3.8/site-packages/prefect/futures.py", line 237, in _result
2023-04-21T15:16:05.120091377Z return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
2023-04-21T15:16:05.120097177Z File "/usr/local/lib/python3.8/site-packages/prefect/states.py", line 100, in _get_state_result
2023-04-21T15:16:05.120103360Z raise MissingResult(
2023-04-21T15:16:05.120110136Z prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
2023-04-21T15:16:05.141538481Z 15:16:05.140 | INFO | prefect.engine - Engine execution of flow run '5c996762-8edc-4584-9f70-4d1d6adf2b3b' aborted by orchestrator: This run has already terminated.
Zanie
04/21/2023, 6:06 PMTask run ‘generate_operation_id-a9e17ed9-0’ - Task run ‘43baff18-8292-4cd7-a694-2dcf6d1a0c77’ already finished.Indicates that the task run id already exists and has a terminal state
Ben Ayers-Glassey
04/21/2023, 7:46 PMDeceivious
04/21/2023, 8:45 PMZanie
04/21/2023, 8:47 PMBen Ayers-Glassey
04/21/2023, 8:50 PMZanie
04/21/2023, 8:51 PMBen Ayers-Glassey
04/21/2023, 8:51 PM20:44:50.907 | INFO | prefect.agent - Submitting flow run 'ddf9a230-7086-4b7e-bdbf-4474d8a9ce66'
20:44:51.139 | INFO | prefect.agent - Completed submission of flow run 'ddf9a230-7086-4b7e-bdbf-4474d8a9ce66'
20:44:51.160 | INFO | prefect.infrastructure.kubernetes-job - Job 'cerulean-skink-j694d': Pod has status 'Pending'.
20:44:52.475 | INFO | prefect.infrastructure.kubernetes-job - Job 'cerulean-skink-j694d': Pod has status 'Running'.
20:45:01.102 | ERROR | prefect.infrastructure.kubernetes-job - Job 'cerulean-skink-j694d': Job did not complete within timeout of 5s.
20:45:01.138 | INFO | prefect.agent - Reported flow run 'ddf9a230-7086-4b7e-bdbf-4474d8a9ce66' as crashed: Flow run infrastructure exited with non-zero status code -1.
pod_watch_timeout_seconds: null
but didn't fix.job_watch_timeout_seconds: null
but didn't fix.Zanie
04/21/2023, 8:53 PMBen Ayers-Glassey
04/21/2023, 8:54 PMZanie
04/21/2023, 8:54 PMBen Ayers-Glassey
04/21/2023, 8:55 PMZanie
04/21/2023, 8:56 PMprefect deployment inspect
or prefect block inspect
for the relevant objects show a value for that field?Ben Ayers-Glassey
04/21/2023, 9:38 PMprefect deployment apply
? Is that a known kind of issue people have encountered in the past? Or is it possible I'm doing something wrong?###
### DO NOT EDIT BELOW THIS LINE
###
...and we are totally editing below that line.
It worked until now, but maybe the time has come where we shouldn't do that anymore.
So I guess... I need to look at our "infrastructure block", and see if I can change the job_watch_timeout_seconds
seconds there.Dekel R
05/04/2023, 1:58 PM