I tried writing a SQLiteConnection ResourceManager as a demo Prefect Community #ask-community

I tried writing a SQLiteConnection ResourceManager...

xyzy

04/09/2021, 4:04 PM

I tried writing a SQLiteConnection ResourceManager as a demo, but when executing with DaskExecutor it leads to problems with having different threads:

Copy code

ProgrammingError('SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 8032 and this is thread id 2308.')

Is there a way to work around this other than using a LocalExecutor? e.g. a parameter for the ResourceManager to tell it to keep everything that uses it in the same thread?

Dylan

04/09/2021, 4:25 PM

Hey Klaus, taking a look 👍

Dylan

04/09/2021, 5:41 PM

Hey @xyzy I don’t think it’s possible to specify that because any executor that utilizes parallelism allows these kinds of constraints generally

Dylan

04/09/2021, 5:42 PM

It’s possible that object just isn’t thread-safe and switching to processes in your Dask Executor would fix the issue?

Dylan

04/09/2021, 5:42 PM

Further, make sure that the objects produced by your ResourceManager are serializable

Dylan

04/09/2021, 5:42 PM

https://github.com/PrefectHQ/prefect/pull/3139

Dylan

04/09/2021, 5:49 PM

May be relevant ☝️

xyzy

04/09/2021, 5:54 PM

ok, thanks for looking into this!

xyzy

04/09/2021, 5:55 PM

will probably give up the idea of using resource managers for db access

Dylan

04/09/2021, 5:56 PM

DB connection objects are tricky to share

Dylan

04/09/2021, 5:57 PM

If you have a short reproducible example it would be great to add it to the issue above

xyzy

04/09/2021, 6:46 PM

sorry, I don't have anything

xyzy

04/09/2021, 6:46 PM

instead of the resource manager approach I switched to a simple contextmanager based helper function that is called within a task

4 Views

Open in Slack

Previous Next