Hi, while trying to execute a task that connects to a db(mongo) and extracts a collection I get this error :

Unexpected error: TypeError("cannot pickle '_thread.lock' object")

The task blueprint looks like this :

def mongo_connect():    
    db = get_db() // wrapper on top of pymongoclient to connect to the specified db
    collection = get_collection(db). // wrapper on top of pymongoclient to get a specified collection form the db
    return collection

I understand that prefect needs to pickle all the tasks , but this code works fine while being run as an independent script (without prefect decorators). How can I make the connection threadlock safe or rather pickle safe?

Hey @Abhas P, Prefect pickled all task outputs because that’s a requirement to send it between Dask workers. collection here is still a connection so that can’t be pickled. If you don’t use that, the workaround would be to store your Flow as a script so it doesn’t get serialized.

Could you elaborate on what you are referring to by "if you don't use that" ? Also how can I store the flow as a script ?

Sorry. If you don’t use Dask, The script based storage docs are here

Thanks! So just to reiterate : 1. If I want to use dask - there is no workaround for the db connection pickle issue. 2. If I use any other executor : I can store the flow as a script and make the connection work.

Yeah exactly that is a dask requirement.

Would you be able to point me to a more descriptive example for bitbucket storage? (this resource specifies the basic args , but not a complete example). Again, thank for helping me with this

Nothing beyond the docs here . If you try it and run into an error, I can help with that