we just (yesterday?) started running into a cloudpickle error when starting flow runs:

prefect.engine.signals.FAIL: Flow finished in state <Failed: "Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  AttributeError("\'Comment\' object has no attribute \'_end\'")')">

the flows are stored in GCS, python is 3.9.10 both locally (MacOS) and remote (docker+k8s). same prefect+cloudpickle versions. I've had the same thing happen before when python versions don't match but bumping has always fixed it, not really sure what else to look into now. any suggestions...?

Hey Brett, will definitely raise. Just checking the

Comment

is a custom class?

not ours no. some cloudpickle internal? this was the closest thing I could find online, also features

Comment

https://github.com/pytorch/pytorch/issues/59777

Ah I guess you could avoid this for now by storing as a script? That seems like it could be a version mismatch of a specific library then that uses that as an internal right?

Brett, this is a stupid question, but did you update the Docker image tag on your Kubernetes run to reflect the latest image? Can you share or DM your Storage and run config just to cross-check? Dockerfile may also be useful if you use one

yes definitely using the updated image I think just forcing a re-register (i.e. removing

idempotency_key=flow.serialized_hash()

) might have fixed; I'll probably add some version strings to the hash to prevent this collision in the future. still puzzled though that no version mismatch errors are bubbling up

do your register via prefect register CLI or flow.register()?