statement? If we want to perform multiple i/o operations to & from database what is the way to do that? What is the recommended way to open the connection at the beginning and close it at the end?
m
Mike Logaciuk
02/08/2024, 10:33 AM
We use our own internal code to manage connections to databases using SQLAlchemy and yes: we open the connection per each task. When the data is fetched or exported, the method has try, except, finally transaction that closes the connection each time the method on connection object is executed.
a
Abhishek Mitra
02/08/2024, 11:28 AM
Okay. But doesn't it affect performance?
m
Mike Logaciuk
02/08/2024, 12:12 PM
In terms of single query or insert? Nope, we only close connection and open new in another task to ensure to not have any zombie sessions.
The most limiting aspect are the SQL Servers themselves
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.