https://prefect.io logo
Title
j

James Keegan

05/20/2022, 5:00 PM
cloudpickle
v2.1.0 was just released an hour ago and it has broken the healthcheck when we try to register a flow. We're using the
prefecthq/prefect:0.15.6-python3.7
as a base image. We now get this error when building the docker image.
Step 21/21 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/prefect-dbt-run-modelling.prefect"]' '(3, 7)'
 ---> Running in 1d14c333ced9
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 152, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
AttributeError: Can't get attribute '_make_function' on <module 'cloudpickle.cloudpickle' from '/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle.py'>
I re-ran a successful pipeline from ~4 hours ago to confirm that it wasn't to do with any code changes we made and yeah, it also failed
k

Kevin Kho

05/20/2022, 5:01 PM
Will try to replicate and will create a ticket for it
đź‘Ť 1
z

Zanie

05/20/2022, 5:03 PM
This sounds like mismatched cloudpickle versions
We can’t update the version that’s inside that container. You’ll want to register flows with a matching version.
j

James Keegan

05/20/2022, 5:09 PM
Interesting - so update the version of cloudpickle in the pipeline env to also be 2.1.0?
k

Kevin Kho

05/20/2022, 5:12 PM
I think more like make your registration
cloudpickle
version the same as the one in the
prefecthq/prefect:0.15.6-python3.7
container
I can replicate if my local registration is cloudpickle 2.1 using that same image. cloudpickle version 1.6 worked fine. I think you need to downgrade the registration version
j

James Keegan

05/20/2022, 5:40 PM
Thanks for the advice all - downgrading didn't work, but updating the version to 2.1.0 in the Dockerfile (in an
extra_dockerfile_command
) did! Yeah must have been a version mismatch as stated
i

Ilya Galperin

05/20/2022, 7:10 PM
Thank you everyone in this thread, helped us out 🙂 Specifically adding the following command to our docker storage extra_dockerfile_command per James’ suggestion:
RUN pip install cloudpickle==2.1.0
k

Kyle McChesney

05/20/2022, 8:38 PM
Same issue. Just to confirm, is it the cloud pickle version in server or agent that needs to be the same
z

Zanie

05/20/2022, 8:41 PM
@Kevin Kho let’s open an issue / discourse topic for this so it’s discoverable, I ran into a cloudpickle compat issue today too.
k

Kevin Kho

05/20/2022, 8:42 PM
Base image for Docker storage and flow registration (and probably agent). I will turn this into Discourse