Arsenii02/18/2020, 6:22 AM
a function A over a list L, and then want another function B to continue working on elements of L in a parallel manner after A has finished. Is my understanding correct, that in order to implement this, function A should basically return the same element it got as an input, and pass it to B? i.e. in Python/pseudocode
This seems to work, but I wonder whether this is the most efficient way to do this. I also noticed that, if I have several consecutive `map`s set up in the above manner
def func_A(list_element): do_stuff(list_element) return list_element def func_B(list_element): list_element+=2 with Flow(...): L = Parameter("list", default=[1,2,3]) first_step = func_A.map(L) second_step = func_B.map(first_step)
, it will not begin the next step until the previous step has finished executing for all mapped elements. Is there a way to go around this? Is it because I'm using LocalExecutor and not something like DaskExecutor? Thanks!
A -> B -> C -> D-> ...
Alex Goodman02/18/2020, 3:24 PM
import prefect def func_A(element): return element * 2 def func_B(element): return element + 4 @prefect.task def func_A_B(element): return func_B(func_A(element)) with prefect.Flow("thing") as flow: L = prefect.Parameter("list", default=[1, 2, 3]) result = func_A_B.map(L) flow.run()