Hi Everyone, Zheng Cheng from China here, our use case for Prefect is complex data analysis,
at the beginning we are using ClickHouse and Superset for data visualization,
Our main app is Ruby on Rails, database is PostgreSQL.
ClickHouse just copy everything in PostgreSQL.
And Superset query the ClickHouse to show graph.
As more question asked, our SQL in Superset (aka Virtual Dataset) grow longer and longer, to a point that difficult to manage and reason about.
so we use “dbt” to break these ~100 line SQL into multiple “dbt models”.
But just SQL is still not enough, We need to write Python to do some “data transformation” work.
We want the trigger Python when things happen, so the data visualization on Superset is “near realtime”
We need to use Python to read data from ClickHouse, and write something back to ClickHouse.
We don’t want to use Flask or Django for that. (negative engineering)
After some investigation, we found that Airflow and Luigi is “first generation” tools, they are way too complex.
We don’t care about DAG.
Dagster is kinda early, Prefect seem nice. that’s why I am here.🙂Update: I need to make Python code can be trigger by a HTTP Request, Not sure how to use Prefect to do that
Hello Everyone! Will from North Carolina here, Prefect is wonderful and so is this community! I have a background in Atmospheric Science and all things "Ops". My team has been using Prefect 1.0 for the 6 months to scale out a deep learning factory within the population health space.
Hello folks! Hello Folks!
I'm a researcher, professor, and ML Lead at AI Lab @ Universidade de Brasilia/Brazil. We are reviewing our MLOps stack, and there is a high chance that Prefect is gona replace airflow. It's encouraging to found a very active community. I'm Looking forward to sharing experiences!
Hi y’all! I’m a (almost finished?) phd student in applied math at princeton 🐯 in the beautiful, industrious state of new jersey. Besides enjoying the best pizza and bagels you can find, I do a lot of data analysis and machine learning work that involves quickly spinning up experiments and firing them off onto various computing clusters. I’m looking to incorporate prefect into my workflow so I don’t have to worry about orchestrating the etl parts of my math research! shoutout to @Nate for the help so far and pointing out this slack!
Hello all. I'm a retired engineer turned programming junkie. Read an article on Medium.io today about Prefect and I'm hooked. I've banged my head against the wall for months trying to figure out how to implement AF as a one man band. Took me about 10 minutes to realize the beauty of Prefect.
I have just demonstrated results of my PoC to my team. I have got green light to replace our Talend jobs (they are docker containers) with Prefect2.0 code.
So at the beginning we will just replace Talend's containers with Prefect containers, so it's transparent for other parts of our system.
After that we will see if / how we can use other Prefect features, like Clound and kubernetes deployment.
Thank you all for your help and have a nice day
Hello Everyone,My name is Emil Ordoñez. I work as a Data Engineer at Intrinsic Brands.We are currently starting to implement Prefect as our orchestration tool/platform.We're going to be using it to orchestrate a set of tools containing, but not limited to:• Fivetran ingestions
• Integrator.io flows
• DBT jobs
• ML models built both on R and Python