Kevin Grismore
01/13/2023, 2:47 PMKubernetesJob
block I was using was created from an older version of Prefect, in which Job Watch Timeout Seconds
had a different default value and description. I created a new block and left the default None
. I suspect this may have been related to the jobs appearing as crashed while still running, so I'm running again with the new block.22:03:50.966 | INFO | Task run 'query_es-07038a84-0' - Processed 13249000 records...
22:03:52.824 | INFO | Task run 'query_es-07038a84-0' - Processed 13250000 records...
22:03:57.345 | INFO | Task run 'query_es-07038a84-0' - Processed 13251000 records...
22:35:15.572 | ERROR | prefect.infrastructure.kubernetes-job - Job 'pcv-delivery-sync-and-transform-ktz7t': Job did not complete.
22:35:15.690 | INFO | prefect.agent - Reported flow run '51ee9c5f-6dce-4f64-a660-52b6d069129b' as crashed: Flow run infrastructure exited with non-zero status code -1.
However, the flow logs (and logs from the pod the flow is running in) continue to report successfully:
22:03:50.966 | INFO | Task run 'query_es-07038a84-0' - Processed 13249000 records...
22:03:52.824 | INFO | Task run 'query_es-07038a84-0' - Processed 13250000 records...
22:03:57.345 | INFO | Task run 'query_es-07038a84-0' - Processed 13251000 records...
22:03:59.067 | INFO | Task run 'query_es-07038a84-0' - Processed 13252000 records...
22:03:59.725 | INFO | Task run 'query_es-07038a84-0' - Processed 13253000 records...
22:04:01.140 | INFO | Task run 'query_es-07038a84-0' - Processed 13254000 records...
and so onNone
, it eventually concludes the flow has crashed or otherwise become unreachable.Created task run 'cloud_storage_upload_blob_from_file-9a1bf371-0' for task 'cloud_storage_upload_blob_from_file'
06:52:45 PM
Executing 'cloud_storage_upload_blob_from_file-9a1bf371-0' immediately...
06:52:45 PM
but the task reports as already finished:
Task run '9951b058-0cde-48b3-b7ff-abb0f82a1bc4' already finished.
06:52:46 PM
cloud_storage_upload_blob_from_file-9a1bf371-0
then we get the MissingResult error:
prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
Nate
01/17/2023, 10:53 PMKevin Grismore
01/22/2023, 3:38 PM