https://prefect.io logo
Title
i

Isara Ovin

06/30/2022, 3:04 PM
Hi guys is there any way i can produce the following using prefect
for col in date_columns:
        if col in df_columns:
            df[col] = df[col].str.replace('/', '-')
            with mp.Pool(mp.cpu_count()) as pool:
                df[col] = pool.map(get_parsed_date, df[col])
k

Khuyen Tran

06/30/2022, 3:06 PM
I assume you are using Prefect 1.0?
i

Isara Ovin

06/30/2022, 3:07 PM
yes
k

Khuyen Tran

06/30/2022, 3:19 PM
So you are having trouble with running for loop and using the if-else statement in a flow?
i

Isara Ovin

06/30/2022, 3:22 PM
no, im wondering how i can replicate the processing done with multiprocessing (mp) on a single dataframe for multiple columns in parallel
k

Khuyen Tran

06/30/2022, 3:23 PM
@Anna Geller Maybe you know this better than me
a

Anna Geller

07/01/2022, 1:19 AM
This looks more like a Pandas question more than Prefect, perhaps you may try on Stack overflow? In Prefect 1.0 you would use mapping and you wouldn't need to worry about this low level setup
i

Isara Ovin

07/01/2022, 9:46 AM
ok, tried with mapping but it doesn’t give me the same performance but thank u for responding
🙌 1