Marwan Sarieddine
04/26/2022, 1:28 PM{'_schema': 'Invalid data type: None'}
twice over the last week over our many flow runs.
It seems other folks have encountered this due to a version mismatch between the agent and their execution environment.
However that is not the case for us - additionally the same flow run will proceed to run successfully for future runs without any changes from our end.
See more details in the threadFailed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'})
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
task_run_info = self.client.get_task_run_info(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1406, in get_task_run_info
state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
File "/usr/local/lib/python3.8/site-packages/prefect/engine/state.py", line 390, in deserialize
state = StateSchema().load(json_blob)
File "/usr/local/lib/python3.8/site-packages/marshmallow_oneofschema/one_of_schema.py", line 153, in load
raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}
This caused the task run to hang in a pending state, and subsequently caused the flow run to hang in a running state for 11 hoursKevin Kho
Marwan Sarieddine
04/26/2022, 2:58 PMZanie
Marwan Sarieddine
04/26/2022, 4:25 PMZanie
Marwan Sarieddine
04/26/2022, 5:41 PMget_flow_run_info
File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 209, in run
flow_run_state = client.get_flow_run_info(flow_run_id).state
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1168, in get_flow_run_info
state=State.deserialize(result.serialized_state),
File "/usr/local/lib/python3.8/site-packages/prefect/engine/state.py", line 390, in deserialize
state = StateSchema().load(json_blob)
File "/usr/local/lib/python3.8/site-packages/marshmallow_oneofschema/one_of_schema.py", line 153, in load
raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}
State.deserialize
in the FlowRunInfoResult.__init__
call would be needed if we want to rely on a client-side fixZanie
Marwan Sarieddine
04/29/2022, 8:01 PMZanie
master
a couple days ago for flow/task run info methods.Marwan Sarieddine
04/29/2022, 8:18 PMZanie
Marwan Sarieddine
04/29/2022, 8:20 PM