https://prefect.io logo
Title
a

Aiden Price

09/27/2019, 5:04 AM
Quick question, can you
task.map()
over a Pandas DataFrame? So you could get each row as a Series for each task call. Or does it only work with generic iterables? Thank you.
c

Chris White

09/27/2019, 5:24 AM
I just ran this:
import pandas as pd
from prefect import task, Flow
@task
def return_df():
    return pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]})
@task
def map_df(val):
    print(val)
with Flow("pd-test") as flow:
    output = map_df.map(return_df)
flow_state = flow.run()
and it errored out; I believe we make some implicit internal assumptions that we are mapping over a list. We hope to revisit our mapping implementation in the near future based on user feedback though, so I’ll keep this use case in mind!
If you could chunk your dataframe into a list within that first task, you might be able to make it work, but whether that’s a good solution depends on your individual use case
a

Aiden Price

09/27/2019, 5:28 AM
Sounds like a good plan. Thank you for getting back to me so quickly.
👍 1