https://prefect.io logo
Title
z

Zheng Cheng

06/23/2022, 1:33 PM
Hi Everyone, Zheng Cheng from China here, our use case for Prefect is complex data analysis, at the beginning we are using ClickHouse and Superset for data visualization, Our main app is Ruby on Rails, database is PostgreSQL. ClickHouse just copy everything in PostgreSQL. And Superset query the ClickHouse to show graph. As more question asked, our SQL in Superset (aka Virtual Dataset) grow longer and longer, to a point that difficult to manage and reason about. so we use “dbt” to break these ~100 line SQL into multiple “dbt models”. But just SQL is still not enough, We need to write Python to do some “data transformation” work. We want the trigger Python when things happen, so the data visualization on Superset is “near realtime” We need to use Python to read data from ClickHouse, and write something back to ClickHouse. We don’t want to use Flask or Django for that. (negative engineering) After some investigation, we found that Airflow and Luigi is “first generation” tools, they are way too complex. We don’t care about DAG. Dagster is kinda early, Prefect seem nice. that’s why I am here. 🙂 Update: I need to make Python code can be trigger by a HTTP Request, Not sure how to use Prefect to do that
👋 17
k

Kevin Kho

06/23/2022, 1:44 PM
Hi @Zheng Cheng, yes I think Prefect fits this use case. Welcome to the community!
❤️ 1
👍 1
j

jawnsy

06/23/2022, 1:49 PM
Welcome to the community @Zheng Cheng! 👋
🚀 1
😄 1
z

Zheng Cheng

06/23/2022, 1:50 PM
Thank you haha 😄
m

Matt Conger

06/23/2022, 2:36 PM
Welcome to Prefect @Zheng Cheng!
😄 1
l

Laurie Hindes

06/23/2022, 3:14 PM
Welcome, @Zheng Cheng!
👋 1
😄 1
a

Aimee McManus

06/23/2022, 3:47 PM
Welcome aboard @Zheng Cheng! 👋 Love your profile pic lol
👏 1
😄 1
c

Charleigh Liang

06/24/2022, 2:30 PM
Welcome to the community @Zheng Cheng! Love the cutting edge stack you're building, so excited Prefect is part of it!
😄 1
❤️ 1
k

Khuyen Tran

06/24/2022, 3:07 PM
Sounds like you are in just the right place @Zheng Cheng. Very happy that you chose us over other tools and have a good application for it
😄 1
🚀 1