I of course have a question slightly smiling face All of my

Question

I of course have a question slightly smiling face All of my so called ETLs look the same but it doesn t match the tempaltes I can find in the doc I have a task returning a list of file paths == gt extract I have a task transforming these files in a list of dict for each dic I have a task to load I just want to be able to map the transform to each file then map each returned dic to its load task However I can t find a way to return only the relevant list to the relevant load task

Hugues-Yanis Amanieu · Answer

Something like that `with Flow My ETL as flow ` `list of files = extract ` `dic1 list dic2 list dic3 list = transform map list of files ` `load type1 map dic1 list ` `load type2 map dic2 list ` `load type3 map dic3 list `

Hugues-Yanis Amanieu · Answer

did I miss something somewhere

Dylan · Answer

Hi < Hugues Yanis Amanieu> Since Prefect is deferring computation `transform map` is actually returning a task result So pulling the result apart doesn t work quite that way

Dylan · Answer

If you re looking for some more advanced control flow try <https docs prefect io core task library control flow html switch>

Dylan · Answer

I think that s the best way to accomplish what you re looking for actually

Dylan · Answer

Otherwise I d recommend just passing all of the results to all three load tasks which as you ve noted is inefficient and is what you re looking to prevent

Hugues-Yanis Amanieu · Answer

Yup I was either thinking of making a single big load function or passing the whole result to each load function Not very elegant Not sure how the Switch will help me however Perhaps FilterTask is what I m looking for

Hugues-Yanis Amanieu · Answer

Thanks by the way for the prompt response

Dylan · Answer

Of course