Hey community I m trying to set up Prefect with GreatExpecta

Question

Hey community I m trying to set up Prefect with GreatExpectations using the official task I m trying to run a validation on an in memory Dataframe result of previous task but have trouble to set it up correctly I m trying to use v3 API of GE set up the expectation suite checkpoint according to <https docs greatexpectations io docs guides validation checkpoints how to pass an in memory dataframe to a checkpoint |the guide> until context run checkpoint but struggle to pass said dataframe Would anyone be able to offer some guidance

Kevin Kho · Answer

I don t think you pass an in memory dataframe for that task You point to a DataFrame in storage and run the checkpoint against it

Kevin Kho · Answer

I think would you need to modify the task to support that

Tomek Florek · Answer

Thanks for getting back Kevin What do you mean by DataFrame in storage A saved csv parquet file

Kevin Kho · Answer

Yeah exactly

Tomek Florek · Answer

Alright that makes things much easier I guess I was trying to complicate my life for no big reason Thanks Kevin ok hand will run with that and let know how it went

Tomek Florek · Answer

All went well slightly smiling face

Kevin Kho · Answer

Nice Thanks for circling back