Quick question. I know that .map() can be used as a single for-loop in a flow. Is there a way to use .map() to create a nested for-loop though?
k
Kevin Kho
08/04/2021, 1:33 PM
Hey @Samuel Tober, the quick answer is no. The first map already parallelizes across available resources so the nested map would not add anything.
s
Samuel Tober
08/04/2021, 1:47 PM
Ok! What is the recommended way for doing this?
k
Kevin Kho
08/04/2021, 1:53 PM
Maybe put the loop inside the task like this?
Copy code
from prefect import task, Flow
@task
def abc(x):
return [x, x+1]
@task
def bcd(y):
return y + 1
@task
def cde(z):
# z is two elements so we would want a map
# but we cant here
out = []
for _ in z:
bcd.run(_)
out.append(_)
return out
with Flow("a") as flow:
items = [1,2,3,4]
a = abc.map(items)
b = cde.map(a)
flow.run()
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.