I of course have a question :slightly_smiling_face...
# ask-communityh
Hugues-Yanis Amanieu
08/13/2020, 3:09 PMI of course have a question 🙂 . All of my so-called ETLs look the same, but it doesn't match the tempaltes I can find in the doc.
• I have a task returning a list of file paths ==> extract
• I have a task transforming these files in a list of dict
• for each dic I have a task to load
I just want to be able to map the transform to each file, then map each returned dic to its load task. However I can't find a way to return only the relevant list to the relevant load task
đź‘€ 1
Hugues-Yanis Amanieu
08/13/2020, 3:14 PMSomething like that:
with Flow('My ETL') as flow:list_of_files = extract()dic1_list, dic2_list, dic3_list = transform.map(list_of_files)load_type1.map(dic1_list)load_type2.map(dic2_list)load_type3.map(dic3_list)Hugues-Yanis Amanieu
08/13/2020, 3:15 PMdid I miss something somewhere?
d
Dylan
08/13/2020, 3:22 PM
Hi @Hugues-Yanis Amanieu!
Since Prefect is deferring computation,
transform.mapDylan
08/13/2020, 3:22 PM
If you’re looking for some more advanced control flow, try: https://docs.prefect.io/core/task_library/control_flow.html#switch
Dylan
08/13/2020, 3:23 PM
I think that’s the best way to accomplish what you’re looking for, actually
Dylan
08/13/2020, 3:25 PM
Otherwise, I’d recommend just passing all of the results to all three load tasks (which, as you’ve noted, is inefficient and is what you’re looking to prevent)
h
Hugues-Yanis Amanieu
08/13/2020, 3:28 PMYup I was either thinking of making a single big load function or passing the whole result to each load function. Not very elegant...
Not sure how the Switch will help me however. Perhaps FilterTask is what I'm looking for.
Hugues-Yanis Amanieu
08/13/2020, 3:29 PMThanks by the way for the prompt response
d
Dylan
08/13/2020, 3:29 PM
Of course!