• redsquare

    redsquare

    3 months ago
    Hey - Steve from sunny Manchester, UK here - I am looking to replace our current airflow implementation with Prefect
    redsquare
    Kevin Kho
    +7
    9 replies
    Copy to Clipboard
  • Zheng Cheng

    Zheng Cheng

    3 months ago
    Hi Everyone, Zheng Cheng from China here, our use case for Prefect is complex data analysis, at the beginning we are using ClickHouse and Superset for data visualization, Our main app is Ruby on Rails, database is PostgreSQL. ClickHouse just copy everything in PostgreSQL. And Superset query the ClickHouse to show graph. As more question asked, our SQL in Superset (aka Virtual Dataset) grow longer and longer, to a point that difficult to manage and reason about. so we use “dbt” to break these ~100 line SQL into multiple “dbt models”. But just SQL is still not enough, We need to write Python to do some “data transformation” work. We want the trigger Python when things happen, so the data visualization on Superset is “near realtime” We need to use Python to read data from ClickHouse, and write something back to ClickHouse. We don’t want to use Flask or Django for that. (negative engineering) After some investigation, we found that Airflow and Luigi is “first generation” tools, they are way too complex. We don’t care about DAG. Dagster is kinda early, Prefect seem nice. that’s why I am here.🙂 Update: I need to make Python code can be trigger by a HTTP Request, Not sure how to use Prefect to do that
    Zheng Cheng
    Kevin Kho
    +5
    8 replies
    Copy to Clipboard
  • William Clark

    William Clark

    3 months ago
    Hello Everyone! Will from North Carolina here, Prefect is wonderful and so is this community! I have a background in Atmospheric Science and all things "Ops". My team has been using Prefect 1.0 for the 6 months to scale out a deep learning factory within the population health space.
    William Clark
    Kevin Kho
    +4
    6 replies
    Copy to Clipboard
  • r

    Rodrigo Mont’Alegre

    2 months ago
    Hi, Rodrigo here. I'm currently evaluating whether to use airflow or prefect to orchestrate data pipelines.
    r
    j
    +8
    10 replies
    Copy to Clipboard
  • Thinam Tamang

    Thinam Tamang

    2 months ago
    Hello everyone 👋, I am Thinam Tamang. I am an NLP Developer and I have been learning Prefect 2.0. It's great to connect with you all here.
    Thinam Tamang
    Nate
    +7
    10 replies
    Copy to Clipboard
  • f

    Fabricio Braz

    2 months ago
    Hello folks! Hello Folks! I'm a researcher, professor, and ML Lead at AI Lab @ Universidade de Brasilia/Brazil. We are reviewing our MLOps stack, and there is a high chance that Prefect is gona replace airflow. It's encouraging to found a very active community. I'm Looking forward to sharing experiences! Best wishes!
    f
    Chris Reuter
    +10
    12 replies
    Copy to Clipboard
  • Dev Dabke

    Dev Dabke

    2 months ago
    Hi y’all! I’m a (almost finished?) phd student in applied math at princeton 🐯 in the beautiful, industrious state of new jersey. Besides enjoying the best pizza and bagels you can find, I do a lot of data analysis and machine learning work that involves quickly spinning up experiments and firing them off onto various computing clusters. I’m looking to incorporate prefect into my workflow so I don’t have to worry about orchestrating the etl parts of my math research! shoutout to @Nate for the help so far and pointing out this slack!
    Dev Dabke
    Bianca Hoch
    +6
    11 replies
    Copy to Clipboard
  • b

    BarnBuilder

    2 months ago
    Hello all. I'm a retired engineer turned programming junkie. Read an article on Medium.io today about Prefect and I'm hooked. I've banged my head against the wall for months trying to figure out how to implement AF as a one man band. Took me about 10 minutes to realize the beauty of Prefect.
    b
    j
    +7
    9 replies
    Copy to Clipboard
  • m

    Marcin Grzybowski

    2 months ago
    Hi all, I have just demonstrated results of my PoC to my team. I have got green light to replace our Talend jobs (they are docker containers) with Prefect2.0 code. So at the beginning we will just replace Talend's containers with Prefect containers, so it's transparent for other parts of our system. After that we will see if / how we can use other Prefect features, like Clound and kubernetes deployment. Thank you all for your help and have a nice day
    m
    Anna Geller
    +7
    9 replies
    Copy to Clipboard
  • Emil Ordoñez

    Emil Ordoñez

    2 months ago
    Hello Everyone, My name is Emil Ordoñez. I work as a Data Engineer at Intrinsic Brands. We are currently starting to implement Prefect as our orchestration tool/platform. We're going to be using it to orchestrate a set of tools containing, but not limited to: • Fivetran ingestions • Integrator.io flows • DBT jobs • ML models built both on R and Python
    Emil Ordoñez
    j
    +8
    12 replies
    Copy to Clipboard