Some of my prefect tasks need to use a database connection. Connections are not serializable, so tasks would have to construct one on run, but this is pricy. If you have a bunch of tasks that require the connection is there a way to provide them with something like a connection pool?
k
Kevin Kho
06/10/2022, 8:17 PM
Are you using Dask? If you use Dask, then you absolutely need to serialize because that’s how Dask passes data. If you aren’t using Dask, you can try turning off the serialization:
Copy code
@task(checkpoint=False)
def get_client():
....
a
Aaron Goebel
06/10/2022, 8:23 PM
Oh great. These tasks don't need dask, but the results a couple tasks down are executed on dask
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.