Getting this whenever I try to run a Flow with a M...
# prefect-community
m
Getting this whenever I try to run a Flow with a Mapped task on Prefect Server:
Copy code
Failed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'},)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/cloud/task_runner.py", line 193, in initialize_run
    map_index=map_index,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1387, in get_task_run_info
    state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/state.py", line 362, in deserialize
    state = StateSchema().load(json_blob)
  File "/usr/local/lib/python3.6/site-packages/marshmallow_oneofschema/one_of_schema.py", line 144, in load
    raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}
Using 0.13.19 right now Looks like some kind of serialization problem?
d
Hi @matta! Make sure that your Prefect version for your Flow isn’t ahead of your Prefect Server instance. That’s usually the cause of errors like this
m
@Dylan Thanks! I double-checked with the Interactive API and turns out it was actually running 0.13.17. I changed the version in the flow, though now it's failing to build? Any idea what the problem might be here?
Copy code
File "/usr/local/lib/python3.6/dist-packages/prefect/client/client.py", line 771, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/usr/local/lib/python3.6/dist-packages/prefect/core/flow.py", line 1462, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 359, in build
    self._build_image(push=push)
  File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 423, in _build_image
    self._parse_generator_output(output)
  File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 661, in _parse_generator_output
    or parsed.get("errorDetail", {}).get("message")
AttributeError: 'NoneType' object has no attribute 'strip'
d
Looks like something related to your Flow storage. Would you mind sharing your Flow configuration code?
m
So this stuff from within the Flow itself?
Copy code
local_dir_path = path.dirname(path.realpath(__file__))

storage = Docker(
    registry_url=getenv("REGISTRY_URL"),
    image_name=getenv("IMAGE_NAME"),
    image_tag=getenv("IMAGE_TAG"),
    python_dependencies=[
        "cryptography",
        "snowflake-connector-python",
        "toolz",
        "sqlalchemy",
        "snowflake-sqlalchemy",
        "mailchimp-marketing",
    ],
    build_kwargs={"network_mode": "host"},
)


environment = KubernetesJobEnvironment(
    job_spec_file=f"{local_dir_path}/job_spec.yaml", unique_job_name=True
)
There's also an
azure-pipelines.yml
and a
job_spec.yaml
d
Is the Prefect version on the machine attempting to register the flow running the same version as your Server instance?
m
Oh, I'm not sure! How would I check that?
d
Copy code
prefect version
with the cli will do it 👍
What are you using to manage your python environments?
m
I commit to the repo and then an Azure Pipeline builds the Flow.
Using pip and docker containers.
I'll add a call to
prefect version
to the script that adds the pip calls
okay, came out as 0.13.17
d
Hmmmmm
Everything in that configuration looks fine to me
Can you run
prefect diagnostics
on that machine?
m
Not totally sure how! I guess I could add it to the script that runs the pip calls?
I'll try that.
d
I might try building this flow on your laptop to make sure the process is working the way you expect, then moving back to the CI process once that’s working
m
So like have a local install of Prefect Server?
That'd probably be useful either way!
@Dylan This is what Prefect Diagnostics gave me
Copy code
2021-01-11T22:43:27.9828069Z {
2021-01-11T22:43:27.9829026Z   "config_overrides": {},
2021-01-11T22:43:27.9829651Z   "env_vars": [],
2021-01-11T22:43:27.9830310Z   "system_information": {
2021-01-11T22:43:27.9831927Z     "platform": "Linux-4.14.203-156.332.amzn2.x86_64-x86_64-with-Ubuntu-18.04-bionic",
2021-01-11T22:43:27.9832830Z     "prefect_backend": "server",
2021-01-11T22:43:27.9833474Z     "prefect_version": "0.13.17",
2021-01-11T22:43:27.9834118Z     "python_version": "3.6.9"
2021-01-11T22:43:27.9834746Z   }
2021-01-11T22:43:27.9835199Z }
d
You should be able to register your flow without a Prefect server instance
m
Everything's supposed to go through CI/CD though, not sure if I'm allowed to bypass the Azure Pipeline.
d
That does make sense
I’d make sure your local environment is matching the CI/CD exactly and if you still experience an error open an issue
m
Do you think upgrading might fix anything? At least from 0.13.17 -> 0.13.19?
Not sure I'd quite be able to replicate the work cluster on my machine, but I'll see!
d
Definitely worth a shot!
I’d recommend getting to 0.14.x as soon as you can as well
A lot of great stuff in the new releases!
m
Yeah, my dev environment is using 0.14!
Gonna lean on Ops to try and persuade them to upgrade.
@Dylan Huzzah, got them to upgrade 0.13.19, and now the flows are building again! But mapped tasks are now giving this error again:
Copy code
Failed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'},)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/cloud/task_runner.py", line 193, in initialize_run
    map_index=map_index,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1387, in get_task_run_info
    state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/state.py", line 362, in deserialize
    state = StateSchema().load(json_blob)
  File "/usr/local/lib/python3.6/site-packages/marshmallow_oneofschema/one_of_schema.py", line 144, in load
    raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}
Hrm, so
prefect version
says
0.13.19
, but when I use the Interactive API it says
0.13.17
full output:
Copy code
{
  "data": {
    "api": {
      "backend": "SERVER",
      "mode": "normal",
      "version": "2020-11-29",
      "core_version": "0.13.17+20.g4d0c8d8c6",
      "release_timestamp": "2020-11-30T03:08:05Z"
    }
  }
}
d
Are you calling
prefect version
on the same machine as your server instance?
m
Yep! Ops fixed it, btw - turns out there were different Prefect versions on different internal components! 🙂
Thanks for your help! The suggestion to go in and run
prefect version
was how I was able to discover the mismatched versions.