https://prefect.io logo
Title
b

Brian McFeeley

07/25/2019, 5:43 PM
If my tasks need to make some writes/updates to a database, are there any recommendations for managing connections to prevent contention and not creating a new connection per invocation? looking at https://docs.prefect.io/guide/core_concepts/tasks.html#overview I wonder if it might be a reasonable idea to create one connection per task as state that lives inside the task. We've done that with celery in the past. Does that make sense?
c

Chris White

07/25/2019, 5:49 PM
As long as your design of the connection adheres to https://docs.prefect.io/guide/tutorials/task-guide.html#be-careful-with-task-attributes that could be reasonable; a few additional possibilities: - you could use a state handler to throttle DB connections and prevent too many from opening at once - use a connection pool on your database, and have your tasks access connection strings from that pool