Hi Everyone, Zheng Cheng from China here, our use case for Prefect is complex data analysis,
at the beginning we are using ClickHouse and Superset for data visualization,
Our main app is Ruby on Rails, database is PostgreSQL.
ClickHouse just copy everything in PostgreSQL.
And Superset query the ClickHouse to show graph.
As more question asked, our SQL in Superset (aka Virtual Dataset) grow longer and longer, to a point that difficult to manage and reason about.
so we use “dbt” to break these ~100 line SQL into multiple “dbt models”.
But just SQL is still not enough, We need to write Python to do some “data transformation” work.
We want the trigger Python when things happen, so the data visualization on Superset is “near realtime”
We need to use Python to read data from ClickHouse, and write something back to ClickHouse.
We don’t want to use Flask or Django for that. (negative engineering)
After some investigation, we found that Airflow and Luigi is “first generation” tools, they are way too complex.
We don’t care about DAG.
Dagster is kinda early, Prefect seem nice. that’s why I am here.
Update: I need to make Python code can be trigger by a HTTP Request, Not sure how to use Prefect to do that