Hi, I was wondering what people do when coming across parts of objects that are not serializable by Cloudpickle when trying to save some result objects to GCS. The result objects are heavily nested so it is difficult to identify which parts of them are not pickleable. Is it considered poor practice to try to send the whole result object to GCS? Or is there some way around it?

Hey @tkanas, Does using the GCSResult not meet your needs / did you encounter an error with some aspect of your result?

I was wondering what people do when coming across parts of objects that are not serializable by Cloudpickle

Usually it's not that difficult to guess which objects are not serializable, these are the ones with external system access, various sessions, service clients, etc. The only real way to deal with this is to re-create them within each task, as there is no way to transfer, for example, a file pointer.

@Kyle Moon-Wright we were trying to pickle and persist the entire flow result state object so that we could have access to even information on tasks that failed (such as the error messages). From what I have tried,

GCSResult

only saves results of successful tasks.

Ah gotcha - you may have some interest in this open issue and it’s included threads, though the use case is slightly different