https://prefect.io logo
b

Borut Hafner

01/29/2021, 1:29 PM
Hello All, I want to one sqlalchemy engine through the flow, to avoid code duplicaton. My simple solution was to make a task which returns sqlalchemy engine object as result. It works when executing it locally, but returns error when executing it on server( cannot pickle local_object ). Is there a solution to this problem or is there a best practice how to share connection in a flow? Thanks!
🙌 1
👀 2
a

Amanda Wee

01/29/2021, 1:52 PM
I've seen this suggested: https://docs.prefect.io/core/idioms/resource-manager.html My own solution was to use a flow state handler to setup the connection when the flow enters running state, storing it in what is essentially a thread-safe singleton, then disconnect when the flow enters finished state. I'm using the local executor though.
b

Borut Hafner

01/29/2021, 1:56 PM
Thanks!
Got it working with resource_manager.