kwmiebach
09/02/2022, 2:46 PMsqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140193261803264 and this is thread id 140192735962880.
I can also paste the part of the code where sqlite is called.
I am also trying to guess the reason behind the error. There is an sqlite object created in a function. Within the function I define another function which uses this object. But both python functions are not prefect flows or tasks. They live within a bigger prefect flow. So here comes my first question: Does prefect 2 create a different thread for each nested function inside a flow or a task. Otherwise I cannot explain why the 2 parts of the code would run in different threads.Khuyen Tran
09/02/2022, 2:53 PMkwmiebach
09/02/2022, 2:59 PMkwmiebach
09/02/2022, 3:00 PMKhuyen Tran
09/02/2022, 3:01 PMRyan Peden
09/02/2022, 3:03 PMkwmiebach
09/02/2022, 3:04 PMkwmiebach
09/02/2022, 3:04 PMkwmiebach
09/02/2022, 3:05 PMnonlocal
keyword.kwmiebach
09/02/2022, 3:06 PMRyan Peden
09/02/2022, 3:08 PMkwmiebach
09/02/2022, 3:09 PMkwmiebach
09/02/2022, 3:10 PMRyan Peden
09/02/2022, 3:12 PMConcurrentTaskRunner
, which uses a thread pool to run tasks. Your nested function creates a closure around the sqlite3.Row
created on the main thread, so you see this error when you execute the function later on inside a taskkwmiebach
09/02/2022, 3:14 PMSequentialTaskRunner
also uses threads? but in sequence?kwmiebach
09/02/2022, 3:18 PMDB_FIBU
as a normal parameterkwmiebach
09/02/2022, 3:19 PMRyan Peden
09/02/2022, 3:24 PMSeqentialTaskRunner
and see if the problem goes away. Again, it might not be the solution you want to use for your production flow but would provide useful information.kwmiebach
09/02/2022, 3:31 PMkwmiebach
09/02/2022, 3:39 PMkwmiebach
09/02/2022, 3:39 PMkwmiebach
09/02/2022, 3:45 PMSequentialTaskRunner
the whole time now. When I comment the 'task' line, where the pipeline is executed, it works. When I uncomment it, which means the pipeline is executed in a prefect 2 task, then the 'different threads' error re-appears.kwmiebach
09/02/2022, 3:46 PMkwmiebach
09/02/2022, 3:46 PMRyan Peden
09/02/2022, 3:48 PMRyan Peden
09/02/2022, 3:51 PMkwmiebach
09/02/2022, 4:04 PMkwmiebach
09/02/2022, 4:05 PM