https://prefect.io logo
y

Ysabel Caballes

07/30/2023, 12:47 PM
Hello! I'm running into a weird data validation error, would really appreciate some help. I have a flow that looks like this:
Copy code
@flow
def clean_data_flow(df: pd.DataFrame) -> pd.DataFrame:
    df = remove_nulls(df)
    return df
When I try to run it, I get this error:
File "pydantic/validators.py", line 765, in find_validators
`RuntimeError: no validator found for <class 'pandas.core.frame.DataFrame'>, see
arbitrary_types_allowed
in Config` If I remove the data validators, so my flow looks like this:
Copy code
@flow
def clean_data_flow(df):
    df = remove_nulls(df)
    return df
My code works fine. Not sure why this is happening.
d

Daniel

07/31/2023, 10:46 PM
Prefect uses pydantic to validate the parameter data types, and pydantic doesn't have a validator for dataframes. If you were creating your own pydantic model you could use
arbitrary_types_allowed
but since you're using
@flow
I don't think you can do that. You can either skip validation as you have done, or pass a more standard type. If you're going to deploy the flow then the parameters also need to be json-serialisable.
y

Ysabel Caballes

08/02/2023, 1:04 PM
thanks @Daniel, that was very helpful
1