Thread
#prefect-community
    i

    Isara Ovin

    2 months ago
    Hi guys is there any way i can produce the following using prefect
    for col in date_columns:
            if col in df_columns:
                df[col] = df[col].str.replace('/', '-')
                with mp.Pool(mp.cpu_count()) as pool:
                    df[col] = pool.map(get_parsed_date, df[col])
    Khuyen Tran

    Khuyen Tran

    2 months ago
    I assume you are using Prefect 1.0?
    i

    Isara Ovin

    2 months ago
    yes
    Khuyen Tran

    Khuyen Tran

    2 months ago
    So you are having trouble with running for loop and using the if-else statement in a flow?
    i

    Isara Ovin

    2 months ago
    no, im wondering how i can replicate the processing done with multiprocessing (mp) on a single dataframe for multiple columns in parallel
    Khuyen Tran

    Khuyen Tran

    2 months ago
    @Anna Geller Maybe you know this better than me
    Anna Geller

    Anna Geller

    2 months ago
    This looks more like a Pandas question more than Prefect, perhaps you may try on Stack overflow? In Prefect 1.0 you would use mapping and you wouldn't need to worry about this low level setup
    i

    Isara Ovin

    2 months ago
    ok, tried with mapping but it doesn’t give me the same performance but thank u for responding