Max Kureykin

    Max Kureykin

    1 year ago
    Hi everyone! Is it possible to pass SQLAlchemy PG session through tasks? I've tried to create, but faced the pickling problems. Is there any predefined way to handle such stuff(sessions/temp objects)?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Max Kureykin, so the requirement to be serializeable is because that is how Dask sends data to workers. You can get around this by using a LocalExecutor and storing your flow as a script
    Max Kureykin

    Max Kureykin

    1 year ago
    So, will it be possible to use file-stored flows with Dask cluster, not locally?
    Kevin Kho

    Kevin Kho

    1 year ago
    You mean the Flow itself is stored in the Dask cluster right? Is this a separate question or related to the previous? The answer though is that you need an agent running to pick up the Flow, and the Flow needs to be accessible for the agent. Normally you have the Prefect loading in the Flow from somewhere than passing the work on to Dask
    Max Kureykin

    Max Kureykin

    1 year ago
    The problem is that database session initialization task returns object that cannot be pickled, so the execution fails with kinda such errors:
    cannot pickle '_thread.RLock' object
    . I just wanna know if it is possible to avoid, without recreating the session. I've seen examples in Prefect repo, but there are new db session every time. I believe that it's could be done somehow.
    Kevin Kho

    Kevin Kho

    1 year ago
    If you’re using DaskExecutor, this is not a Prefect limitation, but a Dask one. Dask sends work to the workers by using
    cloudpickle
    , so you need to recreate the session on the worker side.
    Max Kureykin

    Max Kureykin

    1 year ago
    Yeah, got it, thanks a lot!
    Kevin Kho

    Kevin Kho

    1 year ago
    No problem!