Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

set the channel description: Discussion about the data ecosystem: from database to analytics, and everything in between.

<https://www.exploreomni.com/blog/user-permissions>

As of today, Omni is now commercially available which is pretty cool - the release coincides with a bunch of product updates as well including new permission features: <https://www.exploreomni.com/blog/user-permissions>

<@UKVFX6N3B> might have a better / more concrete answer for you, but from my perspective the product is much more mature  and performant (it was started by some of the original members of the Looker team).  This is possibly unfair, but I think of Preset / Superset as "open source Looker", and Omni as next generation Looker

I think the data exploration &amp; workbooking experience in Omni is awesome

We're pushing for self-serve analytics, so it's nice to sort of set-and-forget the model and folks can still create custom table calcs/write custom sql or even do custom joins if they know how without tampering with the underlying models

Then you can "promote" certain things from their workbook if they're worthwhile

Interesting, I still haven't found a better product than Periscope which got swallowed by Sisense

I need a product where we can share/embed externally as part of our app with row security. I'll definitely have a look at Omni, agree that Superset feels open source and rough but Preset are polishing it alot

Forgot to add, it is interesting the amount of modern tools that are now powered by Clickhouse (tinybird/posthog etc) also that uber/ebay/cloudflare all moved to it

Interested to hear if anyone is using the likes of Atlan/Transform/Amundsen for democratizing their data (internally)

I’m trying to set up Amundsen in my K8S cluster now. Having a lot of difficulty. So much that I’m considering switching to DataHub instead.

another option is <https://open-metadata.org/|https://open-metadata.org/>, not tried it just been looking into this ecosystem

datahub looks needy <https://www.reddit.com/r/dataengineering/comments/yxrh9y/metadata_store_which_one_to_choose_openmetadata/ix1l98v?utm_medium=android_app&amp;utm_source=share&amp;context=3|https://www.reddit.com/r/dataengineering/comments/yxrh9y/metadata_store_which_one_to_choose_openmetadata/ix1l98v?utm_medium=android_app&amp;utm_source=share&amp;context=3>

Yeah the “prerequisite” chart with all those requirements before you can deploy the DataHub chart with their six services looks pretty scary.

We've been looking at a bunch of solutions but no decisions yet. Atlan and Alation both look interesting, not sure about the free/cheap/open source solutions yet. Should probably have a closer look at Amundsen but curious to hear if anyone else has an opinion.

cheers <@U03RA4JQ9C3>, Alation is a new one to me - yet another one to look at :cold_sweat:

I worry as it seems when you choose your platform in this space it is very hard to back out

Yeah, I have the same worry. We initially looked at 10-12 different vendors and narrowed it down to the ones mentioned, just haven't been able to prioritize doing proof-of-concepts etc. yet, and still want to add a 3rd alternative in the form of something less costly.

duckdb anyone ? I replaced a postgres db (used for OLAP) with duckdb + parquet file

do you use it on a local machine, host it somewhere on a VM and give people access?

lack of clustering an issue with duck, fine for local jobs but not to support a high throughout analytical api behind

That's the common issue I've heard as well - scalability/no options for user/role management

but if you don't care about that and single node is fine then :boom:

<https://clickhouse.com/docs/en/operations/utilities/clickhouse-local/|https://clickhouse.com/docs/en/operations/utilities/clickhouse-local/>

In honor you of <@U03LH98F8S2> :redsquare:

For me, parquet/duck is a way to stage data in a query-able format to check that things are clean before moving to Postgres db.

Retool is my favorite new toy.  Building out workflows where users interact with the data.

In our case, we have nurses doing clinical validation of our AI output and saving their validation and notes back in to the db.

Real human-in-the-loop stuff, without a SE team to build the UI for us.  This is the next gen of BI tools, imo.

I have been looking at Hamilton package to manage feature store as well as feature dependency. It has it's own execution engine as well. Has anyone thought or tried to use it with Prefect?

dbt Cloud has <https://twitter.com/getdbt/status/1631364925397295109?t=IB9YIvEKRiG2SGz5_VCB4Q&amp;s=19|webhooks now>

From the perspective of someone not working in a major US-based company, the costs for some of the tools in the ML/Data Science/Data Engineering space are kinda bonkers tbh.

This may be old news for some, but <https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i|pandas 2 is built on Arrow>. This might be the thing that creates a tipping point for Arrow's adoption. It feels like Wes has been encouraging it for years.

Would be interested to hear how many of you are using custom validation vs out of the box packages, as in <https://dropbox.tech/infrastructure/balancing-quality-and-coverage-with-our-data-validation-framework|this use case>.

For our frontend/app analytics we are using <https://avo.app> for schema curation + code gen to assist validation of events+props entering the data pipeline - it comes with a handy Inspector to aid observability and highlights any events coming through not conforming to schema - our pipeline can choose to bin or dead-letter any events not conforming

trying to use it for back-end but the types are not really sophisticated enough so still a challenge - I have asked avo to pivot and support backend validation too:)

<@U02AMBJ8MS7> what about you/your experiences. I have seen various DBT attempts but for me this is too late in the process and it needs to be caught earlier in any pipeline (at source ideally(

Lots of our users are fond of <https://greatexpectations.io/|Great Expectations> and I've seen their demos - it is really impressive. I believe <@U019RALEAUW> uses <https://www.soda.io/|Soda>. No hands on experience with these myself.

Generally not a fan of putting effort into building a bespoke validation framework (it would be like rolling your own orchestrator :sweat_smile:)

We also use Monte Carlo internally but it isn't OSS

Yeah I could never justify the 'build your own' time/costs

Ale even maintains a <https://pypi.org/project/prefect-soda-core/|collection> for it

I have a similar concept prefect flow that does various checks on clickhouse to report its and the ingestion health- soda-ish, table counts etc etc

Any Hex.Tech/Count.Co/other users in here - hex have just released EU hosting which opens it up to me but was wondering if there is anything better out there?

We're loving hex here -- makes it easy to open up access to our warehouse and share notebooks. The embedding support for tools like notion is also :100:

Where does hex stop and omni take over <@UKVFX6N3B>

In my mind, Omni is for persistent dashboards that can be created by analysts using only sql + config. Hex can be used for the same thing, but the modeling support won't be the same. We'll sometimes see users build a query in Omni using the point-and-click interface and then take that query to hex to use python to perform other transformations / use one-off datasets in comparison

Also depends on the persona :sweat_smile: I have Hex access but I'm just not as fast with it compared to Omni

So Hex lets that user share the notebook, Omni lets that user share a dashboard

The notebook can have things we don't support in Omni, while the dashboard may be easier to get off of the ground depending on the persona

Interesting, thanks for your insights+experience chaps, as ever. Appreciated

Where do you see that Hex has released EU hosting? We've been waiting for that to happen, but can't see anything in their blog.

image.png

Very cool, will send my contact an email. Thank you!

Hi all, doing some research and was curious had anyone used MS SQL server with Prefect before?

we have some flows that connect to ms sql