https://prefect.io logo
Title
b

Brett Naul

03/16/2022, 2:11 PM
we just (yesterday?) started running into a cloudpickle error when starting flow runs:
prefect.engine.signals.FAIL: Flow finished in state <Failed: "Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  AttributeError("\'Comment\' object has no attribute \'_end\'")')">
the flows are stored in GCS, python is 3.9.10 both locally (MacOS) and remote (docker+k8s). same prefect+cloudpickle versions. I've had the same thing happen before when python versions don't match but bumping has always fixed it, not really sure what else to look into now. any suggestions...?
k

Kevin Kho

03/16/2022, 2:13 PM
Hey Brett, will definitely raise. Just checking the
Comment
is a custom class?
b

Brett Naul

03/16/2022, 2:14 PM
not ours no. some cloudpickle internal? this was the closest thing I could find online, also features
Comment
https://github.com/pytorch/pytorch/issues/59777
also tried bumping from cloudpickle 1.6 to 2.0, same error
k

Kevin Kho

03/16/2022, 2:17 PM
Ah I guess you could avoid this for now by storing as a script? That seems like it could be a version mismatch of a specific library then that uses that as an internal right?
:upvote: 1
a

Anna Geller

03/16/2022, 2:40 PM
Brett, this is a stupid question, but did you update the Docker image tag on your Kubernetes run to reflect the latest image? Can you share or DM your Storage and run config just to cross-check? Dockerfile may also be useful if you use one
b

Brett Naul

03/16/2022, 2:42 PM
yes definitely using the updated image I think just forcing a re-register (i.e. removing
idempotency_key=flow.serialized_hash()
) might have fixed; I'll probably add some version strings to the hash to prevent this collision in the future. still puzzled though that no version mismatch errors are bubbling up
a

Anna Geller

03/16/2022, 2:45 PM
do your register via prefect register CLI or flow.register()?
b

Brett Naul

03/16/2022, 4:06 PM
prefect.Client().register(flow, idempotency_key=flow.serialized_hash())
👍 1