hi there, i'm seeing an error:
```RuntimeError: &l...
# ask-communityj
Jai P
03/02/2023, 4:01 PMhi there, i'm seeing an error:
Copy code
RuntimeError: <asyncio.locks.Event object at 0x11447f760 [unset]> is bound to a different event loopJai P
03/02/2023, 4:01 PM Copy code
import random
from prefect import flow, task
from prefect.utilities.asyncutils import run_sync_in_worker_thread
@task
def my_random():
return random.random()
@flow
def my_subflow():
result = my_random()
return result
@flow
async def my_test_flow():
subflows = [run_sync_in_worker_thread(deepcopy(my_subflow)) for _ in range(10)]
results = await asyncio.gather(*subflows)
print(results)
if __name__ == "__main__":
asyncio.run(my_test_flow())prefect cloud loginJai P
03/02/2023, 4:04 PMhere's the full stack trace
Jai P
03/02/2023, 4:42 PMthe tl;dr here is that i have a bunch of subflows i want to start in worker threads on the machine, and i guess i'm just a bit confused the best way to do this
c
Christopher Boyd
03/03/2023, 1:53 PMdo they need to be subflows doing a thing, or can they be tasks?
Generally for distributed processing like this, the suggestion would be Dask or Ray
j
Jai P
03/03/2023, 7:00 PMit does need to be a flow orchestrating a bunch of other (sub)flows of the same type, since each of those is comprised of a bunch of tasks
c
Christopher Boyd
03/03/2023, 7:02 PMcan they not be spawned off deployment create flow run?
Christopher Boyd
03/03/2023, 7:02 PMj
Jai P
03/03/2023, 7:03 PMso i tried that, the thing is i want those subflows to run in the same infrastructure that the parent flow is in, and when i use a deployment, i need to have an agent ready to go right?
Jai P
03/03/2023, 7:04 PMi'm running in EKS and the top level flow is in a pod, i don't necessarily want to spin up a lot of other pods for each of those subflows because they're actually quite small. i'll be running thousands of the parent flow though
19 Views