Apologies if this has been discussed elsewhere, bu...
# ask-communitym
Mike O'Connor
08/02/2023, 9:18 AMApologies if this has been discussed elsewhere, but there seems to have been a major shift in Prefect’s behaviour when running asyncio flows and tasks. It is severely impacting our flows (we’re going to have to drop back to Prefect <= v2.8.4).
It appears that Prefect now creates a new thread for each task, which means we cannot do things like sharing connections (or even core asyncio structures such as a semaphore or a queue).
The linked github issue describes the problem well. Does anyone have any workarounds, or can anyone from Prefect provide support (CC @Emil Christensen)
https://github.com/PrefectHQ/prefect/issues/9412
👀 1
Mike O'Connor
08/02/2023, 9:20 AMhere’s an example simple test that demonstrates our problems. Each task ends up on a different event loop.
Copy code
import asyncio
import pytest
from prefect import flow, task
@task
async def task_a():
return id(asyncio.get_event_loop())
@task
async def task_b():
return id(asyncio.get_event_loop())
@flow
async def some_flow():
loop_a = await task_a()
loop_b = await task_b()
return loop_a, loop_b
@pytest.mark.asyncio
async def test_task_event_loops():
loop_a, loop_b = await some_flow()
print(f"{loop_a=}")
print(f"{loop_b=}")
assert loop_a == loop_bMike O'Connor
08/02/2023, 9:25 AMok, i see now this is an intentional change. https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-287
but i fail to see how this is compatible with asyncio oriented workloads.
k
Konrad Maliszewski
08/02/2023, 9:29 AM@Thomas Weatherston what did we do in the end to deal with it?
t
Thomas Weatherston
08/02/2023, 11:40 AMWe ended up having to remove all the async behaviour from our flows and just create clients at the task level instead of at the flow level (kinda defeating the point!).
😭 1
j
Jake Kaplan
08/02/2023, 12:58 PMThere was some work done around moving task run execution to the main thread which I believe would resolve this
🙌 1
Jake Kaplan
08/02/2023, 10:25 PMLooking into getting this back on the radar https://github.com/PrefectHQ/prefect/pull/9855
🙏 1
m
Mike O'Connor
08/21/2023, 1:22 PMAny update on this?
c
Conor
09/02/2023, 8:20 PMas mentioned in the pr, i basically have to maintain a fork of prefect with this pr on the client side in order to use it. Would love to get an answer from prefect team on the appetite to accept a change like this or similar as we otherwise really like prefect and would love to continue to use it for our workflow orchestration needs
Conor
09/07/2023, 6:13 AMJake filed followup here with some additional details on why the proposed PR was not acceptable for mainlining: https://github.com/PrefectHQ/prefect/issues/10627