Hi everyone, We've been running into a problem where each time we call multiple subflows with the

asyncio.gather

command, the first subflow executes correctly, though all of the other subflows will error. I believe the problem is compatibility with the

marvin

library. there seems to be a problem with the

async_utils

. The error message I see consistently is:

Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/marvin/utilities/async_utils.py", line 61, in run_sync
    loop = asyncio.get_running_loop()
RuntimeError: no running event loop

Versions: marvin = "1.5.5" prefect = "2.14.3" Has anyone else in the community encountered this problem?

👋 @Gabe Villasana I've got 👀 on this. Do you have a quick path to reproduce on your end, otherwise can see if I can get something to witness it

Hey @Alexander Azzam, I'm deploying 2 flows using the

serve

method when my workers startup. I invoke flow

A

via the

run_deployment

command. Flow

A

calls subflow

B

via

await asyncio.gather

. Inside subflow

B

, I have a number of tasks which are wrappers around marvin functions. I'm using the

ConcurrentTaskRunner

. Happy to share code details over DMs 🙂

gotcha gotcha, that might be enough for me to reproduce but code always helpful - feel free to 👋 in DMs!

If it's helpful: We just had a user kick off a data pipeline which resulted in 5 subflow invocations. 2 succeeded, 3 failed...

I have not seen that error, but if it is helpful, here is a repo with examples of different subflow patterns. Asyncio gather can be used to have non blocking subflows (or sub-deployments). https://github.com/taylor-curran/prefect-patterns/blob/main/flows/composite_flow_patterns/asyncio_gather_sub_deployments.py Also though, task wrapped sub-deployments can also allow you to achieve concurrent subflow execution: https://github.com/taylor-curran/prefect-patterns/blob/main/flows/composite_flow_patterns/task_wrapped_deployments.py

Thanks for sharing this Taylor! Going to dive in today 🙂

A failing subflow should affect the parent issues have been resolved like this to ensure the state of the parent is coupled with the sate of the child

ye this takes place in k8s