https://prefect.io logo
Title
c

Chris Reuter

03/06/2023, 3:12 PM
Would be interested to hear how many of you are using custom validation vs out of the box packages, as in this use case.
r

redsquare

03/06/2023, 3:17 PM
For our frontend/app analytics we are using https://avo.app for schema curation + code gen to assist validation of events+props entering the data pipeline - it comes with a handy Inspector to aid observability and highlights any events coming through not conforming to schema - our pipeline can choose to bin or dead-letter any events not conforming
🥑 1
🙌 2
trying to use it for back-end but the types are not really sophisticated enough so still a challenge - I have asked avo to pivot and support backend validation too:)
@Chris Reuter what about you/your experiences. I have seen various DBT attempts but for me this is too late in the process and it needs to be caught earlier in any pipeline (at source ideally(
💯 1
c

Chris Reuter

03/06/2023, 4:50 PM
Lots of our users are fond of Great Expectations and I've seen their demos - it is really impressive. I believe @ale uses Soda. No hands on experience with these myself. Generally not a fan of putting effort into building a bespoke validation framework (it would be like rolling your own orchestrator 😅)
We also use Monte Carlo internally but it isn't OSS
r

redsquare

03/06/2023, 4:51 PM
Yeah I could never justify the 'build your own' time/costs
Soda looks interesting, thanks
c

Chris Reuter

03/06/2023, 4:53 PM
Ale even maintains a collection for it
👍 1
👏 1
(I think)
r

redsquare

03/06/2023, 4:59 PM
I have a similar concept prefect flow that does various checks on clickhouse to report its and the ingestion health- soda-ish, table counts etc etc
soda looks cool