@Marvin is there a way to run child or sub-flows in parallel within a parent flow?
m
Marvin
04/03/2024, 12:00 PM
Yes, you can run child or sub-flows in parallel within a parent flow. When you have a flow function called inside the execution of another flow, the primary flow is the "parent" flow, while the flow created within the parent is the "child" flow or "subflow." To run subflows in parallel, you can use AnyIO task groups or
asyncio.gather
, allowing asynchronous subflows to execute concurrently.
For more information on composing flows and running them in parallel, you can refer to Prefect's documentation on composing flows.
m
Matic Pecovnik
04/03/2024, 12:02 PM
can you create an example of that?
n
Nate
04/03/2024, 2:40 PM
Copy code
In [1]: from prefect import flow
In [2]: import asyncio
In [3]: @flow
...: async def work():
...: await asyncio.sleep(5)
...:
In [4]: @flow
...: async def parent():
...: await asyncio.gather(*[work() for _ in range(5)])
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.