Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi just wondering are state hooks also run as tasks in separate threads?

foo_py.py

<https://github.com/PrefectHQ/prefect/blob/0f447919a00b04b422f84ae97f5a97e472f8eefc/src/prefect/engine.py#L2873|yep>

unless its async, then it will just run in the global event loop thread i believe, but you could test with something like above

Yeah it seems async functions still run in a separate event loop.

I have an asyncpg connection pool that raises a "cannot perform operation: another operation is in progress".

Is there way to make tasks run on the same thread by default?

are you creating the connection object outside the hook and partialing it to the hook or something? from the error snippet i can’t tell if your error is about threading or something else about concurrent db access - do you have more of the trace?

Yeah I'm creating the connection outside of the hook and partializing it in.

for nate.py

image.png

I don't want to open a new connection for every hook, but it seems to be sharing between threads that's the issue, maybe not tho idk with prefect.

Does prefect handle it's SQLlite and other db connections differently?

Confirmed they are different threads.
init_wrds_table &lt;_MainThread(MainThread, started 26432)&gt;
log_wrdsfill_event &lt;Thread(GlobalEventLoopThread, started daemon 28792)&gt;

Unless you know of a different way, my workaround idea to retain abstraction is to make a decorator that wraps flows in a logger that is itself a flow.

&gt; Does prefect handle it's SQLlite and other db connections differently?
hmm im not sure I follow - in the SDK we often send user code to worker threads to avoid blocking the async orchestration engine, but the server-side handling of db connection objects is somewhat unrelated

is your main concern with <https://gist.github.com/zzstoatzz/e4316401cb128b27db65f51ce93d620c|creating distinct pools> the overhead?

Pretty much, I don't see why I should spam new sessions. The # of connections I'm allowed to the foreign database is also limited. Creating the logging flow wrapper seems to work fine.
<https://gist.github.com/CircuitCM/c0f8e943ee1d08e937cbd27ac3adda47>