Getting this whenever I try to run a Flow with a Mapped task Prefect Community #ask-community

Getting this whenever I try to run a Flow with a M...

matta

01/09/2021, 2:46 AM

Getting this whenever I try to run a Flow with a Mapped task on Prefect Server:

Copy code

Failed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'},)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/cloud/task_runner.py", line 193, in initialize_run
    map_index=map_index,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1387, in get_task_run_info
    state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/state.py", line 362, in deserialize
    state = StateSchema().load(json_blob)
  File "/usr/local/lib/python3.6/site-packages/marshmallow_oneofschema/one_of_schema.py", line 144, in load
    raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}

Using 0.13.19 right now Looks like some kind of serialization problem?

Dylan

01/11/2021, 2:45 PM

Hi @matta! Make sure that your Prefect version for your Flow isn’t ahead of your Prefect Server instance. That’s usually the cause of errors like this

matta

01/11/2021, 5:40 PM

@Dylan Thanks! I double-checked with the Interactive API and turns out it was actually running 0.13.17. I changed the version in the flow, though now it's failing to build? Any idea what the problem might be here?

Copy code

File "/usr/local/lib/python3.6/dist-packages/prefect/client/client.py", line 771, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/usr/local/lib/python3.6/dist-packages/prefect/core/flow.py", line 1462, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 359, in build
    self._build_image(push=push)
  File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 423, in _build_image
    self._parse_generator_output(output)
  File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 661, in _parse_generator_output
    or parsed.get("errorDetail", {}).get("message")
AttributeError: 'NoneType' object has no attribute 'strip'

Dylan

01/11/2021, 5:44 PM

Looks like something related to your Flow storage. Would you mind sharing your Flow configuration code?

matta

01/11/2021, 5:54 PM

So this stuff from within the Flow itself?

Copy code

local_dir_path = path.dirname(path.realpath(__file__))

storage = Docker(
    registry_url=getenv("REGISTRY_URL"),
    image_name=getenv("IMAGE_NAME"),
    image_tag=getenv("IMAGE_TAG"),
    python_dependencies=[
        "cryptography",
        "snowflake-connector-python",
        "toolz",
        "sqlalchemy",
        "snowflake-sqlalchemy",
        "mailchimp-marketing",
    ],
    build_kwargs={"network_mode": "host"},
)


environment = KubernetesJobEnvironment(
    job_spec_file=f"{local_dir_path}/job_spec.yaml", unique_job_name=True
)

matta

01/11/2021, 5:55 PM

There's also an

azure-pipelines.yml

and a

job_spec.yaml

Dylan

01/11/2021, 6:05 PM

Is the Prefect version on the machine attempting to register the flow running the same version as your Server instance?

matta

01/11/2021, 6:10 PM

Oh, I'm not sure! How would I check that?

Dylan

01/11/2021, 6:13 PM

Copy code

prefect version

Dylan

01/11/2021, 6:13 PM

with the cli will do it 👍

Dylan

01/11/2021, 6:14 PM

What are you using to manage your python environments?

matta

01/11/2021, 6:14 PM

I commit to the repo and then an Azure Pipeline builds the Flow.

matta

01/11/2021, 6:14 PM

Using pip and docker containers.

matta

01/11/2021, 6:15 PM

I'll add a call to

prefect version

to the script that adds the pip calls

matta

01/11/2021, 6:18 PM

okay, came out as 0.13.17

Dylan

01/11/2021, 6:45 PM

Hmmmmm

Dylan

01/11/2021, 6:45 PM

Everything in that configuration looks fine to me

Dylan

01/11/2021, 6:48 PM

Can you run

prefect diagnostics

on that machine?

matta

01/11/2021, 7:03 PM

Not totally sure how! I guess I could add it to the script that runs the pip calls?

matta

01/11/2021, 7:03 PM

I'll try that.

Dylan

01/11/2021, 7:20 PM

I might try building this flow on your laptop to make sure the process is working the way you expect, then moving back to the CI process once that’s working

matta

01/11/2021, 7:30 PM

So like have a local install of Prefect Server?

matta

01/11/2021, 7:30 PM

That'd probably be useful either way!

matta

01/11/2021, 10:45 PM

@Dylan This is what Prefect Diagnostics gave me

Copy code

2021-01-11T22:43:27.9828069Z {
2021-01-11T22:43:27.9829026Z   "config_overrides": {},
2021-01-11T22:43:27.9829651Z   "env_vars": [],
2021-01-11T22:43:27.9830310Z   "system_information": {
2021-01-11T22:43:27.9831927Z     "platform": "Linux-4.14.203-156.332.amzn2.x86_64-x86_64-with-Ubuntu-18.04-bionic",
2021-01-11T22:43:27.9832830Z     "prefect_backend": "server",
2021-01-11T22:43:27.9833474Z     "prefect_version": "0.13.17",
2021-01-11T22:43:27.9834118Z     "python_version": "3.6.9"
2021-01-11T22:43:27.9834746Z   }
2021-01-11T22:43:27.9835199Z }

Dylan

01/11/2021, 10:47 PM

You should be able to register your flow without a Prefect server instance

matta

01/11/2021, 10:48 PM

Everything's supposed to go through CI/CD though, not sure if I'm allowed to bypass the Azure Pipeline.

Dylan

01/11/2021, 10:49 PM

That does make sense

Dylan

01/11/2021, 10:49 PM

I’d make sure your local environment is matching the CI/CD exactly and if you still experience an error open an issue

matta

01/11/2021, 10:50 PM

Do you think upgrading might fix anything? At least from 0.13.17 -> 0.13.19?

matta

01/11/2021, 10:56 PM

Not sure I'd quite be able to replicate the work cluster on my machine, but I'll see!

Dylan

01/11/2021, 11:17 PM

Definitely worth a shot!

Dylan

01/11/2021, 11:18 PM

I’d recommend getting to 0.14.x as soon as you can as well

Dylan

01/11/2021, 11:18 PM

A lot of great stuff in the new releases!

matta

01/11/2021, 11:18 PM

Yeah, my dev environment is using 0.14!

matta

01/11/2021, 11:19 PM

Gonna lean on Ops to try and persuade them to upgrade.

matta

01/11/2021, 11:37 PM

@Dylan Huzzah, got them to upgrade 0.13.19, and now the flows are building again! But mapped tasks are now giving this error again:

Copy code

Failed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'},)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/cloud/task_runner.py", line 193, in initialize_run
    map_index=map_index,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1387, in get_task_run_info
    state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/state.py", line 362, in deserialize
    state = StateSchema().load(json_blob)
  File "/usr/local/lib/python3.6/site-packages/marshmallow_oneofschema/one_of_schema.py", line 144, in load
    raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}

matta

01/12/2021, 1:09 AM

Hrm, so

prefect version

says

0.13.19

, but when I use the Interactive API it says

0.13.17

full output:

Copy code

{
  "data": {
    "api": {
      "backend": "SERVER",
      "mode": "normal",
      "version": "2020-11-29",
      "core_version": "0.13.17+20.g4d0c8d8c6",
      "release_timestamp": "2020-11-30T03:08:05Z"
    }
  }
}

Dylan

01/12/2021, 6:27 PM

Are you calling

prefect version

on the same machine as your server instance?

matta

01/12/2021, 7:34 PM

Yep! Ops fixed it, btw - turns out there were different Prefect versions on different internal components! 🙂

matta

01/12/2021, 7:35 PM

Thanks for your help! The suggestion to go in and run

prefect version

was how I was able to discover the mismatched versions.

4 Views

Open in Slack

Previous Next