Thread
#prefect-community
    t

    tiz.io

    2 years ago
    I've just converted an event-based ETL job to Prefect. ATM it just kicks off the job and exits immediately. I'd love to see PIN 14 implemented so Prefect can observe the events and mark the job done when the appropriate event is received.
    Kyle Moon-Wright

    Kyle Moon-Wright

    2 years ago
    Hey @tiz.io, Noted! Good to see the PIN 14 squad speaking out. Can you share about your use case at all? Which aspects would benefit you the most?
    t

    tiz.io

    2 years ago
    You bet. We have a bunch of pre-existing Node scripts that run our ETL. They are highly tuned and Python isn't our strong suit, so we don't want to rewrite them all. They were instrumented to be invoked and send telemetry out through websockets. Our Python flows have no problem triggering the events, but asyncio isn't doing the trick to keep the process running just long enough to collect telemetry and then exit. (likely because asyncio is new to us and we're doing it wrong, but unsure)
    Instead, it just runs forever even after the loop is stopped and closed. It seems accomplishing this using event driven flows would avoid all that drama.
    Kyle Moon-Wright

    Kyle Moon-Wright

    2 years ago
    This is insightful, thanks for sharing!
    t

    tiz.io

    2 years ago
    Epilogue: I knew there'd be a workaround if I tried at it enough. Last night after ditching all of the asyncio logic, I just created my own event loop and programmed it to sit and spin while the code doing the work streams updates in over websockets. I now have the best of both worlds: Prefect triggering and monitoring the jobs while the worker code stays in its original language on-prem. I'm still looking forward to PIN14, but this has us completely happy in the meantime.
    Tyler Wanner

    Tyler Wanner

    2 years ago
    that’s a very interesting use case! We love Python but I also love that you found a way to use Prefect’s expertise for orchestration and your own internal Node expertise for your etl tasks
    t

    tiz.io

    2 years ago
    Indeed. This makes Prefect a great tool to combine the data science folk putting out Python model generators while still allowing the top of the funnel to be whatever the dev team needs.