Some of my prefect tasks need to use a database connection. Connections are not serializable, so tasks would have to construct one on run, but this is pricy. If you have a bunch of tasks that require the connection is there a way to provide them with something like a connection pool?
06/10/2022, 8:17 PM
Are you using Dask? If you use Dask, then you absolutely need to serialize because that’s how Dask passes data. If you aren’t using Dask, you can try turning off the serialization:
06/10/2022, 8:23 PM
Oh great. These tasks don't need dask, but the results a couple tasks down are executed on dask