Hi all 👋
Just wondering: are 3rd party tasks (i.e. the Great Expectations validation) compatible with Prefect Orion?
Anyone using this? I’d want to validate a dataframe from a upstream task with GE in a flow…
04/05/2022, 11:06 AM
They are not, but Prefect 2.0 has something way cooler than just a task library - we now have Prefect Collections! They are more powerful since:
• they live in separate repos making them more composable and modular with your custom Prefect setup (also preventing dependency conflicts)
• they contain more than just custom tasks
GE is on the roadmap - check out available collections here
04/05/2022, 11:14 AM
Thanks for the info. Any hack/ bridge to get things running in the meantime. Or are those things plain incompatible?
04/05/2022, 11:55 AM
You are welcome to help contribute to the GE collection if you want to. Or you can check the code in 1.0 and port it over to your custom 2.0 task. Afaik GE validation could be done with a subprocess calling the GE CLI command