https://prefect.io
Join Slack
With Orion, will subflows be mappable so we can run a dynamic amount of subflows based on some task ...
# ask-community
n
With Orion, will subflows be mappable so we can run a dynamic amount of subflows based on some task result? Didn’t find any previous details on this
a
Asynchronous subflows can be run in parallel by using AnyIO task groups or asyncio.gather. - Source
not sure whether you can use mapping for this but I assume since you can run any Python code in your flow, you can do something like this to e.g. run one flow in parallel multiple times with various inputs:
Copy code
from prefect import flow
from prefect.task_runners import DaskTaskRunner


@flow
def subflow(x):
    print(x)


@flow(task_runner=DaskTaskRunner())
def parent():
    [subflow(i) for i in range(10)]


if __name__ == "__main__":
    parent()
n
In asking this I’d actually wish for sequential execution of a dynamic amount of subflows, but parallel with async is also neat. Your example would not enable a dynamic amount of subflow(i) since we can’t create the range on a PrefectFuture
This is possible
Copy code
@flow()
def parentflow(runs: int):
    for _ in range(runs):
        subflow()

parentflow(4)
But what’d be neat is to determine the amount of subflow runs with a preceding task, kind of like this.
Copy code
@flow()
def parentflow():
    runs = determine_run_amount() # this is a task
    for _ in range(runs):
        subflow()

parentflow()
(but doesn’t work since we can’t range on a PrefectFuture object)
a
I think you would need to get the result first and then it should work:
Copy code
@flow()
def parentflow():
    state = determine_run_amount().wait() # this is a task
    runs_nr = state.result()
    for _ in range(runs_nr):
        subflow()

parentflow()
n
Can you do that?! It worked perfectly 🙏 👑 🙇
🙌 1
I’m so excited for the subflow patterns, and this specifically
a
Me too! 🙂
marvin 1
m
I was just looking at a way to do exactly this with non-orion prefect, but it looks like it’s not possible, right?
looks like it might be, I found an example from Nov 22 just now
a
@Marko Mušnjak you can definitely run subflows in the current Prefect, both sequentially and in parallel. LMK if you have any questions about it or need an example
m
Thank you, I figured it out. As usual, just after sending out the question 🙂
👍 1
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by