Hi, a basic question – what’s the recommended way to loop parameters if .map() doesn’t work with function of multiple outputs?
I have a pipeline like this
with Flow('multi-output func') as flow:
dt = Parameter("urls", default=['dt1', 'dt2', 'dt3'])
for dt in dt_range:
pipe(dt)
thank you
k
Kevin Kho
12/15/2021, 11:58 PM
So the reason it doesn’t work is that a map operation with multiple outputs returns a type of
List[Tuple]
while using it for mapping requires a structure like
Tuple[List]
so you need to reshape the outputs with an intermediate task
k
KhTan
12/16/2021, 4:45 PM
thank you so much for your reply! if the multiple outputs are nested dictionaries with different format and length, probably better to not use .map() with intermediate task?
k
Kevin Kho
12/16/2021, 4:48 PM
I think that sounds like a use case for flatten? Not sure though
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.