Hello All,
I want to one sqlalchemy engine through the flow, to avoid code duplicaton. My simple solution was to make a task which returns sqlalchemy engine object as result. It works when executing it locally, but returns error when executing it on server( cannot pickle local_object ).
Is there a solution to this problem or is there a best practice how to share connection in a flow?
Thanks!
🙌 1
👀 2
a
Amanda Wee
01/29/2021, 1:52 PM
I've seen this suggested:
https://docs.prefect.io/core/idioms/resource-manager.html
My own solution was to use a flow state handler to setup the connection when the flow enters running state, storing it in what is essentially a thread-safe singleton, then disconnect when the flow enters finished state. I'm using the local executor though.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.