https://prefect.io logo
h

Hui Zheng

10/13/2020, 9:10 PM
Hello, I am running into a strange failure at the
Cloudpickle serialization check
step of the health-check during flow.register(). Could you help? The error message is below
Copy code
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 147, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
    flows.append(cloudpickle.load(f))
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  [Previous line repeated 994 more times]
RecursionError: maximum recursion depth exceeded

Removing intermediate container 98613182a8f2
The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/opt/prefect/flows/20-06-0-standard-ag-stage.prefect"]' '(3, 8)'' returned a non-zero code: 1
Traceback (most recent call last):
  File "build_and_register.py", line 174, in <module>
    flow.register(
  File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1588, in register
    registered_flow = client.register(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 734, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1439, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 346, in build
    self._build_image(push=push)
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 412, in _build_image
    raise ValueError(
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
Please see more context and the details in the thread.
The code is shown below, please note that the 1st line of code
flow.storage.build(push=True)
also did the health-check and it passed. however, the health-check failed at the second line of code at flow.register()
Copy code
flow.storage.build(push=True)

    flow.register(
        project_name='release_{}_{}'.format(target_release_u, environment),
        # we will manually turn it on, for now.
        set_schedule_active=False
    )
j

Jim Crist-Harif

10/13/2020, 9:17 PM
What happens if you pass
build=False
to
register
?
h

Hui Zheng

10/13/2020, 9:19 PM
The flow.register() was working for us before at the previous commit. During this commit, we made some changes to the flow, and the new flow runs and passed testing locally. We didn’t change the flow build and registering process.
@Jim Crist-Harif I just tried the
build=False
, and it worked!
Copy code
Step 21/21 : RUN python /opt/prefect/healthcheck.py '[]' '(3, 8)'
 ---> Running in e0bb97c170de
Beginning health checks...
System Version check: OK
Cloudpickle serialization check: OK
Result check: OK
Environment dependency check: OK
All health checks passed.
Removing intermediate container e0bb97c170de
 ---> 5684cb902ac8
Successfully built 5684cb902ac8
Successfully tagged <http://us.gcr.io/semios-data-platform/stage/prefect-scheduler/standard_ag:20.06.0|us.gcr.io/semios-data-platform/stage/prefect-scheduler/standard_ag:20.06.0>
[2020-10-13 21:18:35] INFO - prefect.Docker | Pushing image to the registry...
Pushing [==================================================>]  8.192kB/5.222kB
Flow URL: <https://cloud.prefect.io/semios/flow/5562476d-68be-4d28-abfd-9ef05d24b086>
 └── ID: 5562476d-68be-4d28-abfd-9ef05d24b086
 └── Project: release_20_06_0_stage
 └── Labels: ['20.06.0', 'stage']
Thank you, but why is it? It is safe to register() without a build?
j

Jim Crist-Harif

10/13/2020, 9:23 PM
Since you're manually calling
storage.build
, the flow is already built - calling it again in
flow.register
isn't necessary. If you don't have a good reason for manually calling
flow.storage.build()
above, then I'd recommend removing calling
flow.storage.build
entirely and let
flow.register
handle it.
h

Hui Zheng

10/13/2020, 9:26 PM
Interesting, I tried removing build() first step, and still register(build=true) , and the same error happens again. code
Copy code
# flow.storage.build(push=True)

    flow.register(
        project_name='release_{}_{}'.format(target_release_u, environment),
        # we will manually turn it on, for now.
        set_schedule_active=False
    )
error:
Copy code
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 147, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
    flows.append(cloudpickle.load(f))
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  File "/usr/local/lib/python3.8/site-packages/environs/__init__.py", line 334, in __getattr__
    return functools.partial(self.__custom_parsers__[name], self)
  [Previous line repeated 994 more times]
RecursionError: maximum recursion depth exceeded

Removing intermediate container f006268f1451
so it seems the storage build inside register() seems not working ?
j

Jim Crist-Harif

10/13/2020, 9:31 PM
My guess is something about building the same flow twice isn't working. If you can provide a reproducible example that'd be useful. Edit, missed your first comment above. Hmmm, that's interesting. Can you provide a reproducible example?
h

Hui Zheng

10/13/2020, 10:54 PM
Hi Jim, let me think about how to make a reproducible example for this. Not sure how to do it without actually share with you my flow code, container, and prefect cloud set-up.
meanwhile, I ran into something that I think might related to this error. After I deployed the flow using our first solution as below ,
register(build=False)
,
Copy code
flow.storage.build(push=True)

    flow.register(
        project_name='release_{}_{}'.format(target_release_u, environment),
        # we will manually turn it on, for now.
        set_schedule_active=False
        , build=False
    )
Though The flow is deployed to prefect-cloud, it fails when it runs, with the error below
Copy code
[13 Oct 2020 3:38pm]: Failed to load and execute Flow's environment: KeyError('20_06_0_standard_ag_stage')
NOTE: I think
20_06_0_standard_ag_stage
is the flow name. I also noticed that the flow is registered with
environment=local_environment
, which has always been this since very beginning and had been working, but I wonder if it’s correct because our prefect-cloud agent is hosted on k8e cluster.
@Jim Crist-Harif Hi, I found the reason.
it’s about how I use
environs
in my code. Previously, I instantiate
env
inside the task
Copy code
from environs import Env

@task(
    max_retries=2
)
def dbt_build_run_commands(models: list) -> list: 
    env = Env()
    forecast = env.bool('DBT_RUN_FORECAST', False)

with Flow('Data Platform Flow') as flow:
     # define flow ...
It worked fine for all,
flow.storage.build(push=True)
, local-flow-run and
register-to-prefect-cloud
After the change, I instantiate
env
globally outside the task, because I have other tasks that aksi use
env
Copy code
from environs import Env
env = Env()
@task(
    max_retries=2
)
def dbt_build_run_commands(models: list) -> list: 
    forecast = env.bool('DBT_RUN_FORECAST', False)

with Flow('Data Platform Flow') as flow:
     # define flow ...
It
It works for
flow.storage.build(push=True)
and local-flow-run. however, when it comes to
register-to-prefect-cloud
, it failed. If I switch the
env
back to the previous way, it works for all. Why is that? Is it related to
cloudpickle
? P.S., the code above is in a file called
flow.py
the build-and-register code below is in a separate .py file that imports flow from flow.py
Copy code
from flow import flow
flow.environment = LocalEnvironment(labels=labels)
flow.storage = Docker( ... )
flow.register(.. )
j

Jim Crist-Harif

10/15/2020, 9:26 PM
Hmmm, that is odd, and is likely something to do with cloudpickle. Unfortunately without a reproducible example I don't think I'll be able to give more help here.
Provided that
Env
is pickleable, either method should work fine, so I'm surprised at the results you're getting here.
h

Hui Zheng

10/15/2020, 9:28 PM
The interesting thing is that
flow.storage.build(push=True)
works, but the
storage build inside register()
doesn’t
j

Jim Crist-Harif

10/15/2020, 9:29 PM
Yeah, that's super odd - I'd want to debug that locally to see what's going on. Is there any way you can strip any proprietary logic/code out of your flows to make a reproducible example? I'd guess empty functions/classes that don't do anything should still result in the same behavior.
2 Views