https://prefect.io logo
Title
j

Jai P

11/16/2022, 5:27 PM
👋 hullo, prefect 2 question! my team is trying to figure out in what cases you might want to call a task directly, submit a task, or submit a task and wait for the result, and if there's a general recommendation on those types of things? more clarification in thread
1
for example, why would we call:
my_task.submit().result()
when we could just call
my_task()
and if we're calling
my_task()
...why would we do that if
.submit()
is available? instead, and should we even be calling
Overall it's workable, but each one of these definitely results in relatively different code, since
.submit
returns a Future and the other two are the result itself
r

Ryan Peden

11/16/2022, 6:06 PM
Assuming my_task is a regular function and not an async function, I don't think there's a reason you'd want to call
my_task.submit().result()
instead of just
my_task()
. You might want to use
submit
if you having several tasks running concurrently would speed up your flows. So for example, if your tasks are doing something that would let them run concurrently with other tasks - like non-blocking I/O, or calling a library written in C that releases the GIL (like
numpy
) - then you might want to run several tasks at the same time instead of waiting for one to finish before starting the next one. In that case, you might do something like:
from prefect import flow, task
from prefect.utilities.asyncutils import gather

...task code goes here

@flow()
def my_flow():
    futures = [task_1.submit(), task_2.submit(), task_3.submit()]
    # you might also call task_x.map() on an array of data to get back an array of futures 

    # wait for all tasks to finish before proceeding
    # 'results' will be an array containing the results of all 3 tasks
    results = gather(*[f.result for f in futures])

    # run another task that uses the output of the first three tasks
    process_results(results)
j

Jai P

11/17/2022, 4:17 PM
got it, thank you!
👍 1