m

    Mark Williams

    2 years ago
    Im trying to determine the best route for reading in a list of text files using Prefect. Would I use the LOOP signal to loop over the list of files or should I build my flow as pertained to one file that way my flow represents a specific file that was processed?
    i

    itay livni

    2 years ago
    Let me know if that makes sense to you
    m

    Mark Williams

    2 years ago
    Thanks @itay livni. I actually found this shortly after and have started down that path. I have one issue though. I have an ifelse in my flow to test for validity of the file. If it is not valid, go one path. If valid, go the other
    This is my flow so far. I am not sure about the ifelse is going to work.
    i

    itay livni

    2 years ago
    It doesnt look like the
    ifelse
    is implemented correctly. For a prefect
    ifelse
    this is the basic construct
    disambiguate_bool = count_term_senses.check_multiple_senses(num_senses)
        
        # True branch
        num_clusters = count_term_senses.count_num_senses(
            val_cnt_ser, 
            num_sense_method=median
            )
    
        # False branch (working but bad code. unnenesary step)
        unambiguated_definitions_df = copy_df(definitions_df)
    
        ifelse(
            disambiguate_bool, 
            num_clusters, 
            unambiguated_definitions_df
            )
    Your check needs to come before your map
    m

    Mark Williams

    2 years ago
    hmm so I think I could remove th map on the Valid File step. Just make that a step that builds two lists. One of valid files and another fro invalid files
    I think this should do it although I am not quite sure it is as clear on where the process goes after 'Generate File List'