https://prefect.io
Join Slack
:wave: hullo folks! have a general question about flows vs. tasks vs. just regular functions in pref...
# ask-community
j
👋 hullo folks! have a general question about flows vs. tasks vs. just regular functions in prefect 2.0. are there any high level guidelines on where we may want to use each? i'll try to describe a scenario in the thread
discourse 1
👋 2
let's say we have some plain python code:
Copy code
def a():
    ...

def b():
    ...

def c():
    b()

def d():
    b()
    c()
there are a number of scenarios we could structure this where some things are `@flow`s, some things are `@task`s, or there's only one flow at the top and everything else is tasks, or everything is marked as a 
@flow
, etc. etc.
i guess i'm not looking for the optimal solution for the example above, but more "hey i have a function, when might i want to mark it as a task or a flow, and what are the tradeoffs of doing so"
z
A major difference is that flows block execution while tasks are run in the background.
If you call a flow, it will not return until its final state is reached. If you call a task, it returns a future immediately and the return value can be retrieved later.
j
i can trigger multiple flows in parallel though right? but maybe that adds some clarity about how much you want to deeply nest things as flows
a
We have this Discourse page that may help: https://discourse.prefect.io/t/are-there-any-guidelines-when-to-use-flow-vs-task-decora[…]flow-should-be-defined-at-a-task-subflow-vs-flow-level/522 I added Michael's info there. Regarding running flows in parallel, you could do:
Copy code
@flow
async def main_flow():
    parallel_subflows = [subflow_1(), subflow_2(), subflow_3(), subflow_4()]
    await asyncio.gather(*parallel_subflows)
more on that here
z
We’re also likely to add an interface for running flows in parallel without using async in the future.
👍 2
🙌 1
j
amazing! and again, tasks can't call other tasks right? so if i have a situation where i'd want to, say, programmatically call some set of tasks conditionally, it may be better to have those tasks called via some function rather than a subflow, especially if i don't want the blocking behavior
so like:
Copy code
@task
def a():
    ...

@task
def b(something):
    ...

def c():
    a_result = a()
    b_result = b(a_result)

@flow
def d():
    c()
where, if possible, avoid making 
c
a subflow unless its absolutely necessary (maybe using a different taskrunner, or storage, or something)
z
Yep that makes sense
The tasks would still run concurrently, but only within the subflow. They’d all finish before the subflow returns.
upvote 1
j
gotcha gotcha, ok thanks!
7 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by