Hello I am running into a strange failure at the `Cloudpickl Prefect Community #ask-community

Hello, I am running into a strange failure at the ...

Hui Zheng

10/13/2020, 9:10 PM

Hello, I am running into a strange failure at the

Cloudpickle serialization check

step of the health-check during flow.register(). Could you help? The error message is below

Copy code

Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 147, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
    flows.append(cloudpickle.load(f))
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  [Previous line repeated 994 more times]
RecursionError: maximum recursion depth exceeded

Removing intermediate container 98613182a8f2
The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/opt/prefect/flows/20-06-0-standard-ag-stage.prefect"]' '(3, 8)'' returned a non-zero code: 1
Traceback (most recent call last):
  File "build_and_register.py", line 174, in <module>
    flow.register(
  File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1588, in register
    registered_flow = client.register(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 734, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1439, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 346, in build
    self._build_image(push=push)
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 412, in _build_image
    raise ValueError(
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.

Please see more context and the details in the thread.

Hui Zheng

10/13/2020, 9:13 PM

The code is shown below, please note that the 1st line of code

flow.storage.build(push=True)

also did the health-check and it passed. however, the health-check failed at the second line of code at flow.register()

Copy code

flow.storage.build(push=True)

    flow.register(
        project_name='release_{}_{}'.format(target_release_u, environment),
        # we will manually turn it on, for now.
        set_schedule_active=False
    )

Jim Crist-Harif

10/13/2020, 9:17 PM

What happens if you pass

build=False

register

Hui Zheng

10/13/2020, 9:19 PM

The flow.register() was working for us before at the previous commit. During this commit, we made some changes to the flow, and the new flow runs and passed testing locally. We didn’t change the flow build and registering process.

Hui Zheng

10/13/2020, 9:20 PM

@Jim Crist-Harif I just tried the

build=False

, and it worked!

Copy code

Step 21/21 : RUN python /opt/prefect/healthcheck.py '[]' '(3, 8)'
 ---> Running in e0bb97c170de
Beginning health checks...
System Version check: OK
Cloudpickle serialization check: OK
Result check: OK
Environment dependency check: OK
All health checks passed.
Removing intermediate container e0bb97c170de
 ---> 5684cb902ac8
Successfully built 5684cb902ac8
Successfully tagged <http://us.gcr.io/semios-data-platform/stage/prefect-scheduler/standard_ag:20.06.0|us.gcr.io/semios-data-platform/stage/prefect-scheduler/standard_ag:20.06.0>
[2020-10-13 21:18:35] INFO - prefect.Docker | Pushing image to the registry...
Pushing [==================================================>]  8.192kB/5.222kB
Flow URL: <https://cloud.prefect.io/semios/flow/5562476d-68be-4d28-abfd-9ef05d24b086>
 └── ID: 5562476d-68be-4d28-abfd-9ef05d24b086
 └── Project: release_20_06_0_stage
 └── Labels: ['20.06.0', 'stage']

Hui Zheng

10/13/2020, 9:21 PM

Thank you, but why is it? It is safe to register() without a build?

Jim Crist-Harif

10/13/2020, 9:23 PM

Since you're manually calling

storage.build

, the flow is already built - calling it again in

flow.register

isn't necessary. If you don't have a good reason for manually calling

flow.storage.build()

above, then I'd recommend removing calling

flow.storage.build

entirely and let

flow.register

handle it.

Hui Zheng

10/13/2020, 9:26 PM

Interesting, I tried removing build() first step, and still register(build=true) , and the same error happens again. code

Copy code

# flow.storage.build(push=True)

    flow.register(
        project_name='release_{}_{}'.format(target_release_u, environment),
        # we will manually turn it on, for now.
        set_schedule_active=False
    )

error:

Copy code

Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 147, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
    flows.append(cloudpickle.load(f))
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  [Previous line repeated 994 more times]
RecursionError: maximum recursion depth exceeded

Removing intermediate container f006268f1451

Hui Zheng

10/13/2020, 9:27 PM

so it seems the storage build inside register() seems not working ?

Jim Crist-Harif

10/13/2020, 9:31 PM

~~My guess is something about building the same flow twice isn't working. If you can provide a reproducible example that'd be useful.~~ Edit, missed your first comment above. Hmmm, that's interesting. Can you provide a reproducible example?

Hui Zheng

10/13/2020, 10:54 PM

Hi Jim, let me think about how to make a reproducible example for this. Not sure how to do it without actually share with you my flow code, container, and prefect cloud set-up.

Hui Zheng

10/13/2020, 11:02 PM

meanwhile, I ran into something that I think might related to this error. After I deployed the flow using our first solution as below ,

register(build=False)

Copy code

flow.storage.build(push=True)

    flow.register(
        project_name='release_{}_{}'.format(target_release_u, environment),
        # we will manually turn it on, for now.
        set_schedule_active=False
        , build=False
    )

Though The flow is deployed to prefect-cloud, it fails when it runs, with the error below

Copy code

[13 Oct 2020 3:38pm]: Failed to load and execute Flow's environment: KeyError('20_06_0_standard_ag_stage')

NOTE: I think

20_06_0_standard_ag_stage

is the flow name. I also noticed that the flow is registered with

environment=local_environment

, which has always been this since very beginning and had been working, but I wonder if it’s correct because our prefect-cloud agent is hosted on k8e cluster.

Hui Zheng

10/14/2020, 10:12 PM

@Jim Crist-Harif Hi, I found the reason.

Hui Zheng

10/14/2020, 10:23 PM

it’s about how I use

environs

in my code. Previously, I instantiate

env

inside the task

Copy code

from environs import Env

@task(
    max_retries=2
)
def dbt_build_run_commands(models: list) -> list: 
    env = Env()
    forecast = env.bool('DBT_RUN_FORECAST', False)

with Flow('Data Platform Flow') as flow:
     # define flow ...

It worked fine for all,

flow.storage.build(push=True)

, local-flow-run and

register-to-prefect-cloud

After the change, I instantiate

env

globally outside the task, because I have other tasks that aksi use

env

Copy code

from environs import Env
env = Env()
@task(
    max_retries=2
)
def dbt_build_run_commands(models: list) -> list: 
    forecast = env.bool('DBT_RUN_FORECAST', False)

with Flow('Data Platform Flow') as flow:
     # define flow ...
It

It works for

flow.storage.build(push=True)

and local-flow-run. however, when it comes to

register-to-prefect-cloud

, it failed. If I switch the

env

back to the previous way, it works for all. Why is that? Is it related to

cloudpickle

? P.S., the code above is in a file called

flow.py

the build-and-register code below is in a separate .py file that imports flow from flow.py

Copy code

from flow import flow
flow.environment = LocalEnvironment(labels=labels)
flow.storage = Docker( ... )
flow.register(.. )

Jim Crist-Harif

10/15/2020, 9:26 PM

Hmmm, that is odd, and is likely something to do with cloudpickle. Unfortunately without a reproducible example I don't think I'll be able to give more help here.

Jim Crist-Harif

10/15/2020, 9:27 PM

Provided that

Env

is pickleable, either method should work fine, so I'm surprised at the results you're getting here.

Hui Zheng

10/15/2020, 9:28 PM

The interesting thing is that

flow.storage.build(push=True)

works, but the

storage build inside register()

doesn’t

Jim Crist-Harif

10/15/2020, 9:29 PM

Yeah, that's super odd - I'd want to debug that locally to see what's going on. Is there any way you can strip any proprietary logic/code out of your flows to make a reproducible example? I'd guess empty functions/classes that don't do anything should still result in the same behavior.

2 Views

Open in Slack

Previous Next