Hi Prefect team. I am trying to use the great Expectation task in my prefect flow and i cannot mange to get it work with custom expectations that i wrote myself - https://docs.prefect.io/api/latest/tasks/great_expectations.html#rungreatexpectationsvalidation. It doesnt seem to pick them up. Would you know if we can use custom expectations with prefect, and if we can, does it require any additional import in the code? Thank you for your help !

Both for Prefect 1.0 and 2.0, the GE integration works based on checkpoints. This is also the method recommended by GE. So you'd need to define your expectations e.g. in a Jupyter notebook, create a checkpoint, and you can then trigger a run that will validate your data against that checkpoint if you need a more engineering approach without Jupyter, you can try Pandera. For more resources on GE and Pandera, check out resources here https://discourse.prefect.io/tag/data-validation

thank you Anna for your answer. I followed Alex Streed tutorial and the GE documentation and it was working great until i tried to use my own custom expectations. It does not seem to be picked up when i run the validation task (RunGreatExpectationsValidation). You can see below the error I get. I guess I need to import my custom expectations, but I do not know where and how to do it.

File "/usr/local/lib/python3.8/site-packages/great_expectations/validator/validator.py", line 1010, in _generate_metric_dependency_subgraphs_for_each_expectation_configuration
    expectation_impl().get_validation_dependencies(
TypeError: 'NoneType' object is not callable

what do you mean by custom expectations? Did you follow the GE tutorial and created a Jupyter notebook and a checkpoint? does it work when you run the checkpoint without Prefect? you could also try to analyze the task library code for the GE validation task to see how that works let me cc @alex and @Andrew Huang - they may add some helpful info

no i meant i wrote my own validation functions following this documentation: https://docs.greatexpectations.io/docs/guides/expectations/creating_custom_expectations/overview/

I see - this may require a feature request. But it's unlikely we'll add anything to 1.0 task library. Have you tried Prefect 2.0 already? This is where we could add custom expectations

No we are still using prefect 1.0 for now. I will try to find a workaround. Thank you for your help!