Is there a new thing in prefect 2.0 where you can't return Google BigQuery Clients within a task?
I know you can in Prefect 1.0
I've tried different task runners for 2.0 and they all fail
PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.
a
alex
08/17/2022, 4:25 PM
Hey @Sam Garvis! Results from tasks in Prefect 2.0 need to be pickleable. The GCP clients generally aren’t pickleable, which is why you’re seeing this error.
Also, we have some integrations with BigQuery in our prefect-gcp collection that might be able to help you with what you’re trying to accomplish.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.