https://prefect.io logo
Channels
Powered by
m

Mike O'Connor

08/02/2023, 9:18 AM
Apologies if this has been discussed elsewhere, but there seems to have been a major shift in Prefect’s behaviour when running asyncio flows and tasks. It is severely impacting our flows (we’re going to have to drop back to Prefect <= v2.8.4). It appears that Prefect now creates a new thread for each task, which means we cannot do things like sharing connections (or even core asyncio structures such as a semaphore or a queue). The linked github issue describes the problem well. Does anyone have any workarounds, or can anyone from Prefect provide support (CC @Emil Christensen) https://github.com/PrefectHQ/prefect/issues/9412
👀 1
here’s an example simple test that demonstrates our problems. Each task ends up on a different event loop.
Copy code
import asyncio

import pytest
from prefect import flow, task

@task
async def task_a():
    return id(asyncio.get_event_loop())

@task
async def task_b():
    return id(asyncio.get_event_loop())

@flow
async def some_flow():
    loop_a = await task_a()
    loop_b = await task_b()
    return loop_a, loop_b

@pytest.mark.asyncio
async def test_task_event_loops():
    loop_a, loop_b = await some_flow()
    print(f"{loop_a=}")
    print(f"{loop_b=}")
    assert loop_a == loop_b
ok, i see now this is an intentional change. https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-287 but i fail to see how this is compatible with asyncio oriented workloads.
k

Konrad Maliszewski

08/02/2023, 9:29 AM
@Thomas Weatherston what did we do in the end to deal with it?
t

Thomas Weatherston

08/02/2023, 11:40 AM
We ended up having to remove all the async behaviour from our flows and just create clients at the task level instead of at the flow level (kinda defeating the point!).
😭 1
j

Jake Kaplan

08/02/2023, 12:58 PM
There was some work done around moving task run execution to the main thread which I believe would resolve this
🙌 1
Looking into getting this back on the radar https://github.com/PrefectHQ/prefect/pull/9855
🙏 1
m

Mike O'Connor

08/21/2023, 1:22 PM
Any update on this?
c

Conor

09/02/2023, 8:20 PM
as mentioned in the pr, i basically have to maintain a fork of prefect with this pr on the client side in order to use it. Would love to get an answer from prefect team on the appetite to accept a change like this or similar as we otherwise really like prefect and would love to continue to use it for our workflow orchestration needs
Jake filed followup here with some additional details on why the proposed PR was not acceptable for mainlining: https://github.com/PrefectHQ/prefect/issues/10627
Powered by