Hi everyone! Is it possible to pass SQLAlchemy PG session through tasks? I've tried to create, but faced the pickling problems. Is there any predefined way to handle such stuff(sessions/temp objects)?

Hey @Max Kureykin, so the requirement to be serializeable is because that is how Dask sends data to workers. You can get around this by using a LocalExecutor and storing your flow as a script

So, will it be possible to use file-stored flows with Dask cluster, not locally?

You mean the Flow itself is stored in the Dask cluster right? Is this a separate question or related to the previous? The answer though is that you need an agent running to pick up the Flow, and the Flow needs to be accessible for the agent. Normally you have the Prefect loading in the Flow from somewhere than passing the work on to Dask

The problem is that database session initialization task returns object that cannot be pickled, so the execution fails with kinda such errors:

cannot pickle '_thread.RLock' object

. I just wanna know if it is possible to avoid, without recreating the session. I've seen examples in Prefect repo, but there are new db session every time. I believe that it's could be done somehow.

If you’re using DaskExecutor, this is not a Prefect limitation, but a Dask one. Dask sends work to the workers by using

cloudpickle

, so you need to recreate the session on the worker side.

Yeah, got it, thanks a lot!

No problem!