https://prefect.io logo
Title
a

ARun

09/04/2020, 3:00 PM
Hey guys, How do I keep a database connection active through out an entire flow? I keep getting the pickling error when I try to share/use the database connection within tasks
TypeError: cannot pickle '_thread.lock' object
c

Chris White

09/04/2020, 3:03 PM
Right now the best way would be to use a connection pool, but in the very near future we’ll have a more first-class way of re-using connections with this PR: https://github.com/PrefectHQ/prefect/pull/3139
:upvote: 3
a

ARun

09/04/2020, 3:11 PM
Thanks @Chris White is there a way to disable the picking for certain tasks? I tried the checkpoint=False, but it doesn't work
c

Chris White

09/04/2020, 3:19 PM
No - Prefect supports parallelism / distributed computation in a first-class way which requires objects to be shared among processes; the only way to do that for generic python objects is via pickle (or
cloudpickle
in Prefect’s case)
s

Shaun Cutts

09/04/2020, 3:52 PM
@ARun I haven’t tried it, but I assume that
cloudpickle
extends standard pickle methods, and you can wrap what you don’t want to pickle in a class with __setstate__ and __getstate__ defined, which drop or substitute for whatever content you don’t want to pickle. You could probably define these on a Task-derived class itself to customize how it is pickled.