https://prefect.io logo
#prefect-community
Title
# prefect-community
h

Hugues-Yanis Amanieu

08/13/2020, 3:09 PM
I of course have a question 🙂 . All of my so-called ETLs look the same, but it doesn't match the tempaltes I can find in the doc. • I have a task returning a list of file paths ==> extract • I have a task transforming these files in a list of dict • for each dic I have a task to load I just want to be able to map the transform to each file, then map each returned dic to its load task. However I can't find a way to return only the relevant list to the relevant load task
đź‘€ 1
Something like that:
with Flow('My ETL') as flow:
list_of_files = extract()
dic1_list, dic2_list, dic3_list = transform.map(list_of_files)
load_type1.map(dic1_list)
load_type2.map(dic2_list)
load_type3.map(dic3_list)
did I miss something somewhere?
d

Dylan

08/13/2020, 3:22 PM
Hi @Hugues-Yanis Amanieu! Since Prefect is deferring computation,
transform.map
is actually returning a task result. So, pulling the result apart doesn’t work quite that way
If you’re looking for some more advanced control flow, try: https://docs.prefect.io/core/task_library/control_flow.html#switch
I think that’s the best way to accomplish what you’re looking for, actually
Otherwise, I’d recommend just passing all of the results to all three load tasks (which, as you’ve noted, is inefficient and is what you’re looking to prevent)
h

Hugues-Yanis Amanieu

08/13/2020, 3:28 PM
Yup I was either thinking of making a single big load function or passing the whole result to each load function. Not very elegant... Not sure how the Switch will help me however. Perhaps FilterTask is what I'm looking for.
Thanks by the way for the prompt response
d

Dylan

08/13/2020, 3:29 PM
Of course!