Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
m
Marwan Sarieddine
04/26/2022, 1:28 PMHi folks, we have run into this error message:
{'_schema': 'Invalid data type: None'}We are running prefect version 0.14.22 on both our kubernetes agent, and dask executor on EKS.
Here is the full exception traceback from our logs
Failed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'})
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
task_run_info = self.client.get_task_run_info(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1406, in get_task_run_info
state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
File "/usr/local/lib/python3.8/site-packages/prefect/engine/state.py", line 390, in deserialize
state = StateSchema().load(json_blob)
File "/usr/local/lib/python3.8/site-packages/marshmallow_oneofschema/one_of_schema.py", line 153, in load
raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}the same flow ran successfully prior to the failure, and will most likely run successfully after this failure (I will confirm in a bit) - making this a a rare occurrence (happened only 2 times over the past week and we have not seen this before last week)
k
Kevin Kho
04/26/2022, 2:58 PMLooking at this and don’t have any immediate ideas. Will dig around
m
Marwan Sarieddine
04/26/2022, 2:58 PM(thank you)
z
Zanie
04/26/2022, 4:22 PMThis is due to null states in the backend, sometimes they take time to propagate and if you deserialize a task run payload with a null state it will fail.
Are you on Server or Cloud?
m
Marwan Sarieddine
04/26/2022, 4:25 PMOn cloud
z
Zanie
04/26/2022, 4:26 PMThanks! I’ll investigate a fix.
👍 1
https://github.com/PrefectHQ/prefect/pull/5718 is our general approach here, but I’m investigating a server-side fix as well.
m
Marwan Sarieddine
04/26/2022, 5:41 PMI see - thanks for sharing the suggested fix
hmm in the other flow run where we encountered this issue - the failure happened inside
get_flow_run_infoFile "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 209, in run
flow_run_state = client.get_flow_run_info(flow_run_id).state
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1168, in get_flow_run_info
state=State.deserialize(result.serialized_state),
File "/usr/local/lib/python3.8/site-packages/prefect/engine/state.py", line 390, in deserialize
state = StateSchema().load(json_blob)
File "/usr/local/lib/python3.8/site-packages/marshmallow_oneofschema/one_of_schema.py", line 153, in load
raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}so an equivalent patch to
State.deserializeFlowRunInfoResult.__init__z
Zanie
04/26/2022, 6:02 PMYeah that seems prudent
m
Marwan Sarieddine
04/29/2022, 8:01 PM@Zanie sorry to disturb - I hope you are having a good Friday. Are there any updates as to whether a fix was put into place on the server side ? We haven’t experienced any failures since I last reported this 3 days ago on our end
z
Zanie
04/29/2022, 8:16 PMNo server-side fix, we are probably just going to address it client-side for now since we do not get many reports.
A fix was merged into
masterm
Marwan Sarieddine
04/29/2022, 8:18 PMI see - so we would have to move from 0.14.22 to 1.something ?
z
Zanie
04/29/2022, 8:19 PMOur cherry-pick that change yeah
m
Marwan Sarieddine
04/29/2022, 8:20 PMok