https://prefect.io logo
Channels
Powered by
m

matta

01/09/2021, 2:46 AM
Getting this whenever I try to run a Flow with a Mapped task on Prefect Server:
Copy code
Failed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'},)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/cloud/task_runner.py", line 193, in initialize_run
    map_index=map_index,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1387, in get_task_run_info
    state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/state.py", line 362, in deserialize
    state = StateSchema().load(json_blob)
  File "/usr/local/lib/python3.6/site-packages/marshmallow_oneofschema/one_of_schema.py", line 144, in load
    raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}
Using 0.13.19 right now Looks like some kind of serialization problem?
d

Dylan

01/11/2021, 2:45 PM
Hi @matta! Make sure that your Prefect version for your Flow isn’t ahead of your Prefect Server instance. That’s usually the cause of errors like this
m

matta

01/11/2021, 5:40 PM
@Dylan Thanks! I double-checked with the Interactive API and turns out it was actually running 0.13.17. I changed the version in the flow, though now it's failing to build? Any idea what the problem might be here?
Copy code
File "/usr/local/lib/python3.6/dist-packages/prefect/client/client.py", line 771, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/usr/local/lib/python3.6/dist-packages/prefect/core/flow.py", line 1462, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 359, in build
    self._build_image(push=push)
  File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 423, in _build_image
    self._parse_generator_output(output)
  File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 661, in _parse_generator_output
    or parsed.get("errorDetail", {}).get("message")
AttributeError: 'NoneType' object has no attribute 'strip'
d

Dylan

01/11/2021, 5:44 PM
Looks like something related to your Flow storage. Would you mind sharing your Flow configuration code?
m

matta

01/11/2021, 5:54 PM
So this stuff from within the Flow itself?
Copy code
local_dir_path = path.dirname(path.realpath(__file__))

storage = Docker(
    registry_url=getenv("REGISTRY_URL"),
    image_name=getenv("IMAGE_NAME"),
    image_tag=getenv("IMAGE_TAG"),
    python_dependencies=[
        "cryptography",
        "snowflake-connector-python",
        "toolz",
        "sqlalchemy",
        "snowflake-sqlalchemy",
        "mailchimp-marketing",
    ],
    build_kwargs={"network_mode": "host"},
)


environment = KubernetesJobEnvironment(
    job_spec_file=f"{local_dir_path}/job_spec.yaml", unique_job_name=True
)
There's also an 
azure-pipelines.yml
and a 
job_spec.yaml
d

Dylan

01/11/2021, 6:05 PM
Is the Prefect version on the machine attempting to register the flow running the same version as your Server instance?
m

matta

01/11/2021, 6:10 PM
Oh, I'm not sure! How would I check that?
d

Dylan

01/11/2021, 6:13 PM
Copy code
prefect version
with the cli will do it 👍
What are you using to manage your python environments?
m

matta

01/11/2021, 6:14 PM
I commit to the repo and then an Azure Pipeline builds the Flow.
Using pip and docker containers.
I'll add a call to 
prefect version
to the script that adds the pip calls
okay, came out as 0.13.17
d

Dylan

01/11/2021, 6:45 PM
Hmmmmm
Everything in that configuration looks fine to me
Can you run 
prefect diagnostics
on that machine?
m

matta

01/11/2021, 7:03 PM
Not totally sure how! I guess I could add it to the script that runs the pip calls?
I'll try that.
d

Dylan

01/11/2021, 7:20 PM
I might try building this flow on your laptop to make sure the process is working the way you expect, then moving back to the CI process once that’s working
m

matta

01/11/2021, 7:30 PM
So like have a local install of Prefect Server?
That'd probably be useful either way!
@Dylan This is what Prefect Diagnostics gave me
Copy code
2021-01-11T22:43:27.9828069Z {
2021-01-11T22:43:27.9829026Z   "config_overrides": {},
2021-01-11T22:43:27.9829651Z   "env_vars": [],
2021-01-11T22:43:27.9830310Z   "system_information": {
2021-01-11T22:43:27.9831927Z     "platform": "Linux-4.14.203-156.332.amzn2.x86_64-x86_64-with-Ubuntu-18.04-bionic",
2021-01-11T22:43:27.9832830Z     "prefect_backend": "server",
2021-01-11T22:43:27.9833474Z     "prefect_version": "0.13.17",
2021-01-11T22:43:27.9834118Z     "python_version": "3.6.9"
2021-01-11T22:43:27.9834746Z   }
2021-01-11T22:43:27.9835199Z }
d

Dylan

01/11/2021, 10:47 PM
You should be able to register your flow without a Prefect server instance
m

matta

01/11/2021, 10:48 PM
Everything's supposed to go through CI/CD though, not sure if I'm allowed to bypass the Azure Pipeline.
d

Dylan

01/11/2021, 10:49 PM
That does make sense
I’d make sure your local environment is matching the CI/CD exactly and if you still experience an error open an issue
m

matta

01/11/2021, 10:50 PM
Do you think upgrading might fix anything? At least from 0.13.17 -> 0.13.19?
Not sure I'd quite be able to replicate the work cluster on my machine, but I'll see!
d

Dylan

01/11/2021, 11:17 PM
Definitely worth a shot!
I’d recommend getting to 0.14.x as soon as you can as well
A lot of great stuff in the new releases!
m

matta

01/11/2021, 11:18 PM
Yeah, my dev environment is using 0.14!
Gonna lean on Ops to try and persuade them to upgrade.
@Dylan Huzzah, got them to upgrade 0.13.19, and now the flows are building again! But mapped tasks are now giving this error again:
Copy code
Failed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'},)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/cloud/task_runner.py", line 193, in initialize_run
    map_index=map_index,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1387, in get_task_run_info
    state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/state.py", line 362, in deserialize
    state = StateSchema().load(json_blob)
  File "/usr/local/lib/python3.6/site-packages/marshmallow_oneofschema/one_of_schema.py", line 144, in load
    raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}
Hrm, so 
prefect version
says 
0.13.19
, but when I use the Interactive API it says 
0.13.17
full output:
Copy code
{
  "data": {
    "api": {
      "backend": "SERVER",
      "mode": "normal",
      "version": "2020-11-29",
      "core_version": "0.13.17+20.g4d0c8d8c6",
      "release_timestamp": "2020-11-30T03:08:05Z"
    }
  }
}
d

Dylan

01/12/2021, 6:27 PM
Are you calling 
prefect version
on the same machine as your server instance?
m

matta

01/12/2021, 7:34 PM
Yep! Ops fixed it, btw - turns out there were different Prefect versions on different internal components! 🙂
Thanks for your help! The suggestion to go in and run 
prefect version
was how I was able to discover the mismatched versions.
3 Views
Powered by