Im trying to determine the best route for reading ...
# prefect-community
m
Im trying to determine the best route for reading in a list of text files using Prefect. Would I use the LOOP signal to loop over the list of files or should I build my flow as pertained to one file that way my flow represents a specific file that was processed?
i
Let me know if that makes sense to you
m
Thanks @itay livni. I actually found this shortly after and have started down that path. I have one issue though. I have an ifelse in my flow to test for validity of the file. If it is not valid, go one path. If valid, go the other
This is my flow so far. I am not sure about the ifelse is going to work.
i
It doesnt look like the
ifelse
is implemented correctly. For a prefect
ifelse
this is the basic construct
Copy code
disambiguate_bool = count_term_senses.check_multiple_senses(num_senses)
    
    # True branch
    num_clusters = count_term_senses.count_num_senses(
        val_cnt_ser, 
        num_sense_method=median
        )

    # False branch (working but bad code. unnenesary step)
    unambiguated_definitions_df = copy_df(definitions_df)

    ifelse(
        disambiguate_bool, 
        num_clusters, 
        unambiguated_definitions_df
        )
Your check needs to come before your map
m
hmm so I think I could remove th map on the Valid File step. Just make that a step that builds two lists. One of valid files and another fro invalid files
I think this should do it although I am not quite sure it is as clear on where the process goes after 'Generate File List'