Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Has anyone used Great Expectations, how does it compare to other testing frameworks?

Let’s see what the community has to say. Am curious too what people are using.

Great Expectations is quite focused on working in Jupyter notebooks. If you need some resources about alternatives to running GE in a Jupyter notebook, check out:

• Kevin's tutorial on Pandera - <https://youtu.be/SP6WqCRUkNc?t=2249|video> + <https://github.com/kvnkho/demos/blob/main/meetups/mlops-community/pandera-tutorial.ipynb|code>
• My take on GE: <https://betterprogramming.pub/put-a-stop-to-data-swamps-with-event-driven-data-testing-203d9c1be073?sk=8a3aaad76c66bf21aa3d823754260300|blog>

And how about Soda <https://www.soda.io?|https://www.soda.io?> It is new since last year (I think). Never used it, but I have been hearing a lot from it lately…

If you use dbt, then dbt tests is also a valid options. There even is a some sort of port of GE to dbt <https://github.com/calogica/dbt-expectations|https://github.com/calogica/dbt-expectations>

Is there anyway to do a "cascade" drop with to_sql("replace")?

With Pandas? What do you mean by cascade drop?

<@U02AP2C2LDS> I'd say Pandas `to_sql("replace")` is a convenience method if you need to get things into a DB table quickly. But if you already have a table design with foreign keys, you shouldn't use it - you can always use append to load to a temporary table and then execute a MERGE query.