Hello, what is the best way to use global variables, for example database engine objects, in different tasks?
z
Zanie
04/01/2021, 3:28 PM
Hi Zach -- is your database engine object serializable or is it holding a connection?
z
Zach Khorozian
04/02/2021, 2:44 PM
Hi Michael, it’s a postgres connection created with sqlalchemy create_engine, I don’t think it’s serializable
z
Zanie
04/02/2021, 4:14 PM
There isn't a robust pattern for this yet as it's a bit tricky once you're in a distributed setting. There's a bit of WIP that's out of date now (https://github.com/PrefectHQ/prefect/pull/3139) to create a util for this, but for now you're stuck creating the connection in each task. The definition for it can be stored in a global variable but it'll need to be instantiated in each task.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.