Would be interested to hear how many of you are using custom validation vs out of the box packages, as in this use case.
03/06/2023, 3:17 PM
For our frontend/app analytics we are using https://avo.app for schema curation + code gen to assist validation of events+props entering the data pipeline - it comes with a handy Inspector to aid observability and highlights any events coming through not conforming to schema - our pipeline can choose to bin or dead-letter any events not conforming
trying to use it for back-end but the types are not really sophisticated enough so still a challenge - I have asked avo to pivot and support backend validation too:)
@Chris Reuter what about you/your experiences. I have seen various DBT attempts but for me this is too late in the process and it needs to be caught earlier in any pipeline (at source ideally(
03/06/2023, 4:50 PM
Lots of our users are fond of Great Expectations and I've seen their demos - it is really impressive. I believe @ale uses Soda. No hands on experience with these myself.
Generally not a fan of putting effort into building a bespoke validation framework (it would be like rolling your own orchestrator 😅)
We also use Monte Carlo internally but it isn't OSS
03/06/2023, 4:51 PM
Yeah I could never justify the 'build your own' time/costs