we just (yesterday?) started running into a cloudp...
# ask-community
b
we just (yesterday?) started running into a cloudpickle error when starting flow runs:
Copy code
prefect.engine.signals.FAIL: Flow finished in state <Failed: "Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  AttributeError("\'Comment\' object has no attribute \'_end\'")')">
the flows are stored in GCS, python is 3.9.10 both locally (MacOS) and remote (docker+k8s). same prefect+cloudpickle versions. I've had the same thing happen before when python versions don't match but bumping has always fixed it, not really sure what else to look into now. any suggestions...?
k
Hey Brett, will definitely raise. Just checking the
Comment
is a custom class?
b
not ours no. some cloudpickle internal? this was the closest thing I could find online, also features
Comment
https://github.com/pytorch/pytorch/issues/59777
also tried bumping from cloudpickle 1.6 to 2.0, same error
k
Ah I guess you could avoid this for now by storing as a script? That seems like it could be a version mismatch of a specific library then that uses that as an internal right?
upvote 1
a
Brett, this is a stupid question, but did you update the Docker image tag on your Kubernetes run to reflect the latest image? Can you share or DM your Storage and run config just to cross-check? Dockerfile may also be useful if you use one
b
yes definitely using the updated image I think just forcing a re-register (i.e. removing
idempotency_key=flow.serialized_hash()
) might have fixed; I'll probably add some version strings to the hash to prevent this collision in the future. still puzzled though that no version mismatch errors are bubbling up
a
do your register via prefect register CLI or flow.register()?
b
prefect.Client().register(flow, idempotency_key=flow.serialized_hash())
👍 1