Jai P
11/16/2022, 5:27 PMmy_task.submit().result()
when we could just call
my_task()
and if we're calling my_task()
...why would we do that if .submit()
is available?
instead, and should we even be calling.submit
returns a Future and the other two are the result itselfRyan Peden
11/16/2022, 6:06 PMmy_task.submit().result()
instead of just my_task()
.
You might want to use submit
if you having several tasks running concurrently would speed up your flows. So for example, if your tasks are doing something that would let them run concurrently with other tasks - like non-blocking I/O, or calling a library written in C that releases the GIL (like numpy
) - then you might want to run several tasks at the same time instead of waiting for one to finish before starting the next one.
In that case, you might do something like:
from prefect import flow, task
from prefect.utilities.asyncutils import gather
...task code goes here
@flow()
def my_flow():
futures = [task_1.submit(), task_2.submit(), task_3.submit()]
# you might also call task_x.map() on an array of data to get back an array of futures
# wait for all tasks to finish before proceeding
# 'results' will be an array containing the results of all 3 tasks
results = gather(*[f.result for f in futures])
# run another task that uses the output of the first three tasks
process_results(results)
Jai P
11/17/2022, 4:17 PM