Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

prefect_second_pipeline.png

Hi, I have an earlier pipeline set up on prefect ui which runs without a problem. now i intend to register a separate pipeline to the same project but it’s throwing cloudpickle error. could it be due to setup, or any datatype? not using dask executor. thanks.

Hey <@U01N1U05H7H>, this indicates that you are passing something that cannot be pickled between tasks like a Client (database, or boto3)

If you are not using Dask, you can bypass this by using <https://docs.prefect.io/orchestration/flow_config/storage.html#pickle-vs-script-based-storage|script based storage>

thanks <@U01QEJ9PP53>
the last task of the pipe is write to database, which worked fine in the other pipe. in between tasks i’m passing lists and dataframes, could that cause issues?

it also works in ‘flow.run()’ mode, but not in ‘flow.register’ mode

Lists and DataFrames are serializeable so that it normally not the issue. Yes this happens because `register()` serializes the flow by default

The traceback here says you are returning a Google Client somewhere? Or maybe passing it to a Task?

You can move that code inside the task instead

indeed due to a Google Client! thanks and problem fixed