Hi! I have a flow in which I run a bunch of querie...
# prefect-community
Hi! I have a flow in which I run a bunch of queries. I'd like to establish connection to the db only once, but I'm not sure how to do that in Prefect. If I use a task I cannot just put the db object in the context (as per https://docs.prefect.io/api/latest/utilities/context.html#context it is not recommended to mutate context in tasks and even downstream tasks do not see it). I cannot pass around the db connection as it is not serializable. Is there a recommended way to achieve this?
I like to have a function like
, which maintains a singleton connection (only creates a connection if it hasn't already created one before, otherwise returns the created one). This avoids passing the connection around as a task input, instead you call
within your
Note that if your flow executor is distributed to multiple machines OR uses processes as workers, this method creates a connection for each process. There is no way to get around this afaik, database connections cannot and should not be shared between operating system processes. threads can share connections no problem.