https://prefect.io logo
Title
a

Aaron Goebel

06/10/2022, 8:13 PM
Some of my prefect tasks need to use a database connection. Connections are not serializable, so tasks would have to construct one on run, but this is pricy. If you have a bunch of tasks that require the connection is there a way to provide them with something like a connection pool?
k

Kevin Kho

06/10/2022, 8:17 PM
Are you using Dask? If you use Dask, then you absolutely need to serialize because that’s how Dask passes data. If you aren’t using Dask, you can try turning off the serialization:
@task(checkpoint=False)
def get_client():
    ....
a

Aaron Goebel

06/10/2022, 8:23 PM
Oh great. These tasks don't need dask, but the results a couple tasks down are executed on dask