Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Would be interested to hear how many of you are using custom validation vs out of the box packages, as in <https://dropbox.tech/infrastructure/balancing-quality-and-coverage-with-our-data-validation-framework|this use case>.

For our frontend/app analytics we are using <https://avo.app> for schema curation + code gen to assist validation of events+props entering the data pipeline - it comes with a handy Inspector to aid observability and highlights any events coming through not conforming to schema - our pipeline can choose to bin or dead-letter any events not conforming

trying to use it for back-end but the types are not really sophisticated enough so still a challenge - I have asked avo to pivot and support backend validation too:)

<@U02AMBJ8MS7> what about you/your experiences. I have seen various DBT attempts but for me this is too late in the process and it needs to be caught earlier in any pipeline (at source ideally(

Lots of our users are fond of <https://greatexpectations.io/|Great Expectations> and I've seen their demos - it is really impressive. I believe <@U019RALEAUW> uses <https://www.soda.io/|Soda>. No hands on experience with these myself.

Generally not a fan of putting effort into building a bespoke validation framework (it would be like rolling your own orchestrator :sweat_smile:)

We also use Monte Carlo internally but it isn't OSS

Yeah I could never justify the 'build your own' time/costs

Ale even maintains a <https://pypi.org/project/prefect-soda-core/|collection> for it

I have a similar concept prefect flow that does various checks on clickhouse to report its and the ingestion health- soda-ish, table counts etc etc