Hi Everyone, Zheng Cheng from China here, our use case for Prefect is complex data analysis, at the beginning we are using ClickHouse and Superset for data visualization, Our main app is Ruby on Rails, database is PostgreSQL. ClickHouse just copy everything in PostgreSQL. And Superset query the ClickHouse to show graph. As more question asked, our SQL in Superset (aka Virtual Dataset) grow longer and longer, to a point that difficult to manage and reason about. so we use “dbt” to break these ~100 line SQL into multiple “dbt models”. But just SQL is still not enough, We need to write Python to do some “data transformation” work. We want the trigger Python when things happen, so the data visualization on Superset is “near realtime” We need to use Python to read data from ClickHouse, and write something back to ClickHouse. We don’t want to use Flask or Django for that. (negative engineering) After some investigation, we found that Airflow and Luigi is “first generation” tools, they are way too complex. We don’t care about DAG. Dagster is kinda early, Prefect seem nice. that’s why I am here. 🙂 Update: I need to make Python code can be trigger by a HTTP Request, Not sure how to use Prefect to do that

Hi @Zheng Cheng, yes I think Prefect fits this use case. Welcome to the community!

Welcome to the community @Zheng Cheng! 👋

Thank you haha 😄

Welcome to Prefect @Zheng Cheng!

Welcome, @Zheng Cheng!

Welcome aboard @Zheng Cheng! 👋 Love your profile pic lol

Welcome to the community @Zheng Cheng! Love the cutting edge stack you're building, so excited Prefect is part of it!

Sounds like you are in just the right place @Zheng Cheng. Very happy that you chose us over other tools and have a good application for it