Jai P

11/16/2022, 5:27 PM
👋 hullo, prefect 2 question! my team is trying to figure out in what cases you might want to call a task directly, submit a task, or submit a task and wait for the result, and if there's a general recommendation on those types of things? more clarification in thread
for example, why would we call:
when we could just call
and if we're calling
...why would we do that if
is available? instead, and should we even be calling
Overall it's workable, but each one of these definitely results in relatively different code, since
returns a Future and the other two are the result itself

Ryan Peden

11/16/2022, 6:06 PM
Assuming my_task is a regular function and not an async function, I don't think there's a reason you'd want to call
instead of just
. You might want to use
if you having several tasks running concurrently would speed up your flows. So for example, if your tasks are doing something that would let them run concurrently with other tasks - like non-blocking I/O, or calling a library written in C that releases the GIL (like
) - then you might want to run several tasks at the same time instead of waiting for one to finish before starting the next one. In that case, you might do something like:
from prefect import flow, task
from prefect.utilities.asyncutils import gather

...task code goes here

def my_flow():
    futures = [task_1.submit(), task_2.submit(), task_3.submit()]
    # you might also call on an array of data to get back an array of futures 

    # wait for all tasks to finish before proceeding
    # 'results' will be an array containing the results of all 3 tasks
    results = gather(*[f.result for f in futures])

    # run another task that uses the output of the first three tasks

Jai P

11/17/2022, 4:17 PM
got it, thank you!
👍 1