Hello I am running into a strange failure at the `Cloudpickl

Question

Hello I am running into a strange failure at the `Cloudpickle serialization check` step of the health check during flow register Could you help The error message is below ```Beginning health checks System Version check OK Traceback most recent call last File opt prefect healthcheck py line 147 in lt module gt flows = cloudpickle deserialization check flow file paths File opt prefect healthcheck py line 40 in cloudpickle deserialization check flows append cloudpickle load f File usr local lib python3 8 site packages environs init py line 334 in getattr return functools partial self custom parsers name self File usr local lib python3 8 site packages environs init py line 334 in getattr return functools partial self custom parsers name self File usr local lib python3 8 site packages environs init py line 334 in getattr return functools partial self custom parsers name self Previous line repeated 994 more times RecursionError maximum recursion depth exceeded Removing intermediate container 98613182a8f2 The command bin sh c python opt prefect healthcheck py opt prefect flows 20 06 0 standard ag stage prefect 3 8 returned a non zero code 1 Traceback most recent call last File build and register py line 174 in lt module gt flow register File usr local lib python3 8 site packages prefect core flow py line 1588 in register registered flow = client register File usr local lib python3 8 site packages prefect client client py line 734 in register serialized flow = flow serialize build=build type Any File usr local lib python3 8 site packages prefect core flow py line 1439 in serialize storage = self storage build type Optional Storage File usr local lib python3 8 site packages prefect environments storage docker py line 346 in build self build image push=push File usr local lib python3 8 site packages prefect environments storage docker py line 412 in build image raise ValueError ValueError Your docker image failed to build Your flow might have failed one of its deployment health checks please ensure that all necessary files and dependencies have been included ``` Please see more context and the details in the thread

Hui Zheng · Answer

The code is shown below please note that the 1st line of code `flow storage build push=True ` also did the health check and it passed however the health check failed at the second line of code at flow register ``` flow storage build push=True flow register project name= release format target release u environment we will manually turn it on for now set schedule active=False ```

Jim Crist-Harif · Answer

What happens if you pass `build=False` to `register`

Hui Zheng · Answer

The flow register was working for us before at the previous commit During this commit we made some changes to the flow and the new flow runs and passed testing locally We didn t change the flow build and registering process

Hui Zheng · Answer

< Jim Crist Harif> I just tried the `build=False` and it worked ```Step 21 21 RUN python opt prefect healthcheck py 3 8 gt Running in e0bb97c170de Beginning health checks System Version check OK Cloudpickle serialization check OK Result check OK Environment dependency check OK All health checks passed Removing intermediate container e0bb97c170de gt 5684cb902ac8 Successfully built 5684cb902ac8 Successfully tagged 2020 10 13 21 18 35 INFO prefect Docker | Pushing image to the registry Pushing ================================================== gt 8 192kB 5 222kB Flow URL ID 5562476d 68be 4d28 abfd 9ef05d24b086 Project release 20 06 0 stage Labels 20 06 0 stage ```

Hui Zheng · Answer

Thank you but why is it It is safe to register without a build

Jim Crist-Harif · Answer

Since you re manually calling `storage build` the flow is already built calling it again in `flow register` isn t necessary If you don t have a good reason for manually calling `flow storage build ` above then I d recommend removing calling `flow storage build` entirely and let `flow register` handle it

Hui Zheng · Answer

Interesting I tried removing build first step and still register build=true and the same error happens again code ``` flow storage build push=True flow register project name= release format target release u environment we will manually turn it on for now set schedule active=False ``` error ```Beginning health checks System Version check OK Traceback most recent call last File opt prefect healthcheck py line 147 in lt module gt flows = cloudpickle deserialization check flow file paths File opt prefect healthcheck py line 40 in cloudpickle deserialization check flows append cloudpickle load f File usr local lib python3 8 site packages environs init py line 334 in getattr return functools partial self custom parsers name self File usr local lib python3 8 site packages environs init py line 334 in getattr return functools partial self custom parsers name self File usr local lib python3 8 site packages environs init py line 334 in getattr return functools partial self custom parsers name self Previous line repeated 994 more times RecursionError maximum recursion depth exceeded Removing intermediate container f006268f1451```

Hui Zheng · Answer

so it seems the storage build inside register seems not working

Jim Crist-Harif · Answer

~My guess is something about building the same flow twice isn t working If you can provide a reproducible example that d be useful ~ Edit missed your first comment above Hmmm that s interesting Can you provide a reproducible example

Hui Zheng · Answer

Hi Jim let me think about how to make a reproducible example for this Not sure how to do it without actually share with you my flow code container and prefect cloud set up

Hui Zheng · Answer

meanwhile I ran into something that I think might related to this error After I deployed the flow using our first solution as below `register build=False ` ``` flow storage build push=True flow register project name= release format target release u environment we will manually turn it on for now set schedule active=False build=False ``` Though The flow is deployed to prefect cloud it fails when it runs with the error below ``` 13 Oct 2020 3 38pm Failed to load and execute Flow s environment KeyError 20 06 0 standard ag stage ``` NOTE I think `20 06 0 standard ag stage` is the flow name I also noticed that the flow is registered with `environment=local environment` which has always been this since very beginning and had been working but I wonder if it s correct because our prefect cloud agent is hosted on k8e cluster

Hui Zheng · Answer

< Jim Crist Harif> Hi I found the reason

Hui Zheng · Answer

it s about how I use `environs` in my code Previously I instantiate `env` inside the task ```from environs import Env task max retries=2 def dbt build run commands models list gt list env = Env forecast = env bool DBT RUN FORECAST False with Flow Data Platform Flow as flow define flow ``` It worked fine for all `flow storage build push=True ` local flow run and `register to prefect cloud` After the change I instantiate `env` globally outside the task because I have other tasks that aksi use `env` ```from environs import Env env = Env task max retries=2 def dbt build run commands models list gt list forecast = env bool DBT RUN FORECAST False with Flow Data Platform Flow as flow define flow It``` It works for `flow storage build push=True ` and local flow run however when it comes to `register to prefect cloud` it failed If I switch the `env` back to the previous way it works for all Why is that Is it related to `cloudpickle` P S the code above is in a file called `flow py` the build and register code below is in a separate py file that imports flow from flow py ```from flow import flow flow environment = LocalEnvironment labels=labels flow storage = Docker flow register ```

Jim Crist-Harif · Answer

Hmmm that is odd and is likely something to do with cloudpickle Unfortunately without a reproducible example I don t think I ll be able to give more help here

Jim Crist-Harif · Answer

Provided that `Env` is pickleable either method should work fine so I m surprised at the results you re getting here

Hui Zheng · Answer

The interesting thing is that `flow storage build push=True ` works but the `storage build inside register ` doesn t

Jim Crist-Harif · Answer

Yeah that s super odd I d want to debug that locally to see what s going on Is there any way you can strip any proprietary logic code out of your flows to make a reproducible example I d guess empty functions classes that don t do anything should still result in the same behavior