Channels
#prefect-community
Title
m
Marwan Sarieddine
04/26/2022, 1:28 PMHi folks, we have run into this error message:
Copy code
{'_schema': 'Invalid data type: None'}We are running prefect version 0.14.22 on both our kubernetes agent, and dask executor on EKS.
Here is the full exception traceback from our logs
Copy code
Failed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'})
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
task_run_info = self.client.get_task_run_info(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1406, in get_task_run_info
state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
File "/usr/local/lib/python3.8/site-packages/prefect/engine/state.py", line 390, in deserialize
state = StateSchema().load(json_blob)
File "/usr/local/lib/python3.8/site-packages/marshmallow_oneofschema/one_of_schema.py", line 153, in load
raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}the same flow ran successfully prior to the failure, and will most likely run successfully after this failure (I will confirm in a bit) - making this a a rare occurrence (happened only 2 times over the past week and we have not seen this before last week)
k
Kevin Kho
04/26/2022, 2:58 PM
Looking at this and don’t have any immediate ideas. Will dig around
m
Marwan Sarieddine
04/26/2022, 2:58 PM(thank you)
z
Zanie
04/26/2022, 4:22 PM
This is due to null states in the backend, sometimes they take time to propagate and if you deserialize a task run payload with a null state it will fail.
Are you on Server or Cloud?
m
Marwan Sarieddine
04/26/2022, 4:25 PMOn cloud
z
Zanie
04/26/2022, 4:26 PM
Thanks! I’ll investigate a fix.
👍 1
https://github.com/PrefectHQ/prefect/pull/5718 is our general approach here, but I’m investigating a server-side fix as well.
m
Marwan Sarieddine
04/26/2022, 5:41 PMI see - thanks for sharing the suggested fix
hmm in the other flow run where we encountered this issue - the failure happened inside
get_flow_run_info Copy code
File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 209, in run
flow_run_state = client.get_flow_run_info(flow_run_id).state
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1168, in get_flow_run_info
state=State.deserialize(result.serialized_state),
File "/usr/local/lib/python3.8/site-packages/prefect/engine/state.py", line 390, in deserialize
state = StateSchema().load(json_blob)
File "/usr/local/lib/python3.8/site-packages/marshmallow_oneofschema/one_of_schema.py", line 153, in load
raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}so an equivalent patch to
State.deserializeFlowRunInfoResult.__init__z
Zanie
04/26/2022, 6:02 PM
Yeah that seems prudent
m
Marwan Sarieddine
04/29/2022, 8:01 PM@Zanie sorry to disturb - I hope you are having a good Friday. Are there any updates as to whether a fix was put into place on the server side ? We haven’t experienced any failures since I last reported this 3 days ago on our end
z
Zanie
04/29/2022, 8:16 PM
No server-side fix, we are probably just going to address it client-side for now since we do not get many reports.
A fix was merged into
masterm
Marwan Sarieddine
04/29/2022, 8:18 PMI see - so we would have to move from 0.14.22 to 1.something ?
z
Zanie
04/29/2022, 8:19 PM
Our cherry-pick that change yeah
m
Marwan Sarieddine
04/29/2022, 8:20 PMok
18 Views