Thread
#prefect-community
    t

    tkanas

    1 year ago
    Hi, I was wondering what people do when coming across parts of objects that are not serializable by Cloudpickle when trying to save some result objects to GCS. The result objects are heavily nested so it is difficult to identify which parts of them are not pickleable. Is it considered poor practice to try to send the whole result object to GCS? Or is there some way around it?
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    Hey @tkanas, Does using the GCSResult not meet your needs / did you encounter an error with some aspect of your result?
    Saulius Beinorius

    Saulius Beinorius

    1 year ago
    I was wondering what people do when coming across parts of objects that are not serializable by Cloudpickle
    Usually it's not that difficult to guess which objects are not serializable, these are the ones with external system access, various sessions, service clients, etc. The only real way to deal with this is to re-create them within each task, as there is no way to transfer, for example, a file pointer.
    t

    tkanas

    1 year ago
    @Kyle Moon-Wright we were trying to pickle and persist the entire flow result state object so that we could have access to even information on tasks that failed (such as the error messages). From what I have tried,
    GCSResult
    only saves results of successful tasks.
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    Ah gotcha - you may have some interest in this open issue and it’s included threads, though the use case is slightly different