Karthik
05/18/2023, 9:00 AMimport asyncio
from prefect import flow
@flow
async def subflow_1():
print("Subflow 1 started!")
some_other_flow()
# post result ops
@flow
async def subflow_2():
print("Subflow 2 started!")
some_other_flow_2()
# post result ops
@flow
async def subflow_3():
print("Subflow 3 started!")
some_other_flow_3()
# post result ops
@flow
async def subflow_4():
print("Subflow 4 started!")
some_other_flow_4()
# post result ops
@flow
async def main_flow():
parallel_subflows = [subflow_1(), subflow_2(), subflow_3(), subflow_4()]
await asyncio.gather(*parallel_subflows)
if __name__ == "__main__":
main_flow_state = asyncio.run(main_flow())
Now when I deploy this (or run locally), flow runs are created for the 4 subflows, but that’s about it, thereafter it’s all sequential, nothing gets processed parallely. For Eg: subflow_2 run proceeds only after subflow_3(in turn some_other_flow_3) have finished.
So in the end it makes no difference to calling each of these subflows sequentially instead of using asycio
Am I missing something else here? Kindly adviceOuail Bendidi
05/18/2023, 9:45 AMrun_deployment
and run the subflows in different processes (or other infra block)Karthik
05/18/2023, 9:50 AMOuail Bendidi
05/18/2023, 9:52 AM