Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey community! I’m trying to set up Prefect with GreatExpectations, using the official task.  I’m trying to run a validation on an in-memory Dataframe (result of previous task) but have trouble to set it up correctly. I’m trying to use v3 API of GE, set up the expectation suite, checkpoint according to <https://docs.greatexpectations.io/docs/guides/validation/checkpoints/how_to_pass_an_in_memory_dataframe_to_a_checkpoint/|the guide> (until _context.run_checkpoint)_ but struggle to pass said dataframe. Would anyone be able to offer some guidance?

I don’t think you pass an in memory dataframe for that task. You point to a DataFrame in storage and run the checkpoint against it

I think would you need to modify the task to support that.

Thanks for getting back Kevin! What do you mean by DataFrame in storage? A saved csv/parquet file?

Alright, that makes things much easier, I guess I was trying to complicate my life for no big reason. Thanks Kevin :ok_hand: , will run with that and let know how it went.