Thread
#prefect-community
    KhTan

    KhTan

    10 months ago
    Hi, I have an earlier pipeline set up on prefect ui which runs without a problem. now i intend to register a separate pipeline to the same project but it’s throwing cloudpickle error. could it be due to setup, or any datatype? not using dask executor. thanks.
    Kevin Kho

    Kevin Kho

    10 months ago
    Hey @KhTan, this indicates that you are passing something that cannot be pickled between tasks like a Client (database, or boto3)
    If you are not using Dask, you can bypass this by using script based storage
    KhTan

    KhTan

    10 months ago
    thanks @Kevin Kho the last task of the pipe is write to database, which worked fine in the other pipe. in between tasks i’m passing lists and dataframes, could that cause issues?
    it also works in ‘flow.run()’ mode, but not in ‘flow.register’ mode
    Kevin Kho

    Kevin Kho

    10 months ago
    Lists and DataFrames are serializeable so that it normally not the issue. Yes this happens because
    register()
    serializes the flow by default
    The traceback here says you are returning a Google Client somewhere? Or maybe passing it to a Task?
    You can move that code inside the task instead
    KhTan

    KhTan

    10 months ago
    indeed due to a Google Client! thanks and problem fixed