https://prefect.io
Join Slack
Apologies if this has been discussed elsewhere, but there seems to have been a major shift in Prefec...
# ask-community
m
Apologies if this has been discussed elsewhere, but there seems to have been a major shift in Prefect’s behaviour when running asyncio flows and tasks. It is severely impacting our flows (we’re going to have to drop back to Prefect <= v2.8.4). It appears that Prefect now creates a new thread for each task, which means we cannot do things like sharing connections (or even core asyncio structures such as a semaphore or a queue). The linked github issue describes the problem well. Does anyone have any workarounds, or can anyone from Prefect provide support (CC @Emil Christensen) https://github.com/PrefectHQ/prefect/issues/9412
👀 1
here’s an example simple test that demonstrates our problems. Each task ends up on a different event loop.
Copy code
import asyncio

import pytest
from prefect import flow, task

@task
async def task_a():
    return id(asyncio.get_event_loop())

@task
async def task_b():
    return id(asyncio.get_event_loop())

@flow
async def some_flow():
    loop_a = await task_a()
    loop_b = await task_b()
    return loop_a, loop_b

@pytest.mark.asyncio
async def test_task_event_loops():
    loop_a, loop_b = await some_flow()
    print(f"{loop_a=}")
    print(f"{loop_b=}")
    assert loop_a == loop_b
ok, i see now this is an intentional change. https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-287 but i fail to see how this is compatible with asyncio oriented workloads.
k
@Thomas Weatherston what did we do in the end to deal with it?
t
We ended up having to remove all the async behaviour from our flows and just create clients at the task level instead of at the flow level (kinda defeating the point!).
😭 1
j
There was some work done around moving task run execution to the main thread which I believe would resolve this
🙌 1
Looking into getting this back on the radar https://github.com/PrefectHQ/prefect/pull/9855
🙏 1
m
Any update on this?
c
as mentioned in the pr, i basically have to maintain a fork of prefect with this pr on the client side in order to use it. Would love to get an answer from prefect team on the appetite to accept a change like this or similar as we otherwise really like prefect and would love to continue to use it for our workflow orchestration needs
Jake filed followup here with some additional details on why the proposed PR was not acceptable for mainlining: https://github.com/PrefectHQ/prefect/issues/10627
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by