My flow worked fine until I add a GCP Data Catalog and Big Query piece in my code which is helping me fetch some column names from the Data Catalog and update some data in Big Query. Once I added the Data Catalog Piece and Big Query client.

from google.cloud import datacatalog_v1
from google.cloud import bigquery

client = bigquery.Client()
datacatalog_client = datacatalog_v1.DataCatalogClient()

I get an error saying that: Clients have non-trivial state that is local and unpickleable Can someone tell me what does this mean? I'm assuming this has something to do with GCP. I'm using a local service account json file and also importing pubsub modules, I didn't have any issue then.

Hi @Varun Joshi! A couple of questions. Are you returning the Clients from a task? Are you using the DaskExecutor?

Hey @Kevin Kho, thanks for the reply. I'm not using DaskExecutor. Can you please elaborate on 'returning Clients from a task' part? My task function isn't returning anything, it just calls a few more function to fetch some data and push it further

client here could be anything, right? A variable, dataframe, etc etc.

Not quite. If using the DaskExecutor, it has to be serializable . But you aren’t anyway. I don’t have experience but the bigquery Client, but my initial advice would be to see if you can use that inside a Task inside.

Thanks Kevin, I'll take a look 🙂

Seems this person got around by initializing inside a class. No problem!

Sure, will try that!