https://prefect.io
Join Slack
I tried writing a SQLiteConnection ResourceManager as a demo, but when executing with DaskExecutor i...
# ask-community
x
I tried writing a SQLiteConnection ResourceManager as a demo, but when executing with DaskExecutor it leads to problems with having different threads:
Copy code
ProgrammingError('SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 8032 and this is thread id 2308.')
Is there a way to work around this other than using a LocalExecutor? e.g. a parameter for the ResourceManager to tell it to keep everything that uses it in the same thread?
d
Hey Klaus, taking a look 👍
Hey @xyzy I don’t think it’s possible to specify that because any executor that utilizes parallelism allows these kinds of constraints generally
It’s possible that object just isn’t thread-safe and switching to processes in your Dask Executor would fix the issue?
Further, make sure that the objects produced by your ResourceManager are serializable
May be relevant ☝️
x
ok, thanks for looking into this!
will probably give up the idea of using resource managers for db access
d
DB connection objects are tricky to share
If you have a short reproducible example it would be great to add it to the issue above
x
sorry, I don't have anything
instead of the resource manager approach I switched to a simple contextmanager based helper function that is called within a task
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by