Would be interested to hear how many of you are using custom

Question

Would be interested to hear how many of you are using custom validation vs out of the box packages as in <https dropbox tech infrastructure balancing quality and coverage with our data validation framework|this use case>

redsquare · Answer

For our frontend app analytics we are using <https avo app> for schema curation + code gen to assist validation of events+props entering the data pipeline it comes with a handy Inspector to aid observability and highlights any events coming through not conforming to schema our pipeline can choose to bin or dead letter any events not conforming

redsquare · Answer

trying to use it for back end but the types are not really sophisticated enough so still a challenge I have asked avo to pivot and support backend validation too

redsquare · Answer

< Chris Reuter> what about you your experiences I have seen various DBT attempts but for me this is too late in the process and it needs to be caught earlier in any pipeline at source ideally

Chris Reuter · Answer

Lots of our users are fond of and I ve seen their demos it is really impressive I believe < ale> uses No hands on experience with these myself Generally not a fan of putting effort into building a bespoke validation framework it would be like rolling your own orchestrator sweat smile

Chris Reuter · Answer

We also use Monte Carlo internally but it isn t OSS

redsquare · Answer

Yeah I could never justify the build your own time costs

redsquare · Answer

Soda looks interesting thanks

Chris Reuter · Answer

Ale even maintains a <https pypi org project prefect soda core |collection> for it

Chris Reuter · Answer

I think

redsquare · Answer

I have a similar concept prefect flow that does various checks on clickhouse to report its and the ingestion health soda ish table counts etc etc

redsquare · Answer

soda looks cool