Hi guys is there any way i can produce the following using prefect
Copy code
for col in date_columns:
if col in df_columns:
df[col] = df[col].str.replace('/', '-')
with mp.Pool(mp.cpu_count()) as pool:
df[col] = pool.map(get_parsed_date, df[col])
k
Khuyen Tran
06/30/2022, 3:06 PM
I assume you are using Prefect 1.0?
i
Isara Ovin
06/30/2022, 3:07 PM
yes
k
Khuyen Tran
06/30/2022, 3:19 PM
So you are having trouble with running for loop and using the if-else statement in a flow?
i
Isara Ovin
06/30/2022, 3:22 PM
no, im wondering how i can replicate the processing done with multiprocessing (mp) on a single dataframe for multiple columns in parallel
k
Khuyen Tran
06/30/2022, 3:23 PM
@Anna Geller Maybe you know this better than me
a
Anna Geller
07/01/2022, 1:19 AM
This looks more like a Pandas question more than Prefect, perhaps you may try on Stack overflow?
In Prefect 1.0 you would use mapping and you wouldn't need to worry about this low level setup
i
Isara Ovin
07/01/2022, 9:46 AM
ok, tried with mapping but it doesn’t give me the same performance but thank u for responding
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.