I've just converted an event-based ETL job to Prefect. ATM it just kicks off the job and exits immediately. I'd love to see PIN 14 implemented so Prefect can observe the events and mark the job done when the appropriate event is received.

Hey @tiz.io, Noted! Good to see the PIN 14 squad speaking out. Can you share about your use case at all? Which aspects would benefit you the most?

You bet. We have a bunch of pre-existing Node scripts that run our ETL. They are highly tuned and Python isn't our strong suit, so we don't want to rewrite them all. They were instrumented to be invoked and send telemetry out through websockets. Our Python flows have no problem triggering the events, but asyncio isn't doing the trick to keep the process running just long enough to collect telemetry and then exit. (likely because asyncio is new to us and we're doing it wrong, but unsure)

This is insightful, thanks for sharing!

Epilogue: I knew there'd be a workaround if I tried at it enough. Last night after ditching all of the asyncio logic, I just created my own event loop and programmed it to sit and spin while the code doing the work streams updates in over websockets. I now have the best of both worlds: Prefect triggering and monitoring the jobs while the worker code stays in its original language on-prem. I'm still looking forward to PIN14, but this has us completely happy in the meantime.

that’s a very interesting use case! We love Python but I also love that you found a way to use Prefect’s expertise for orchestration and your own internal Node expertise for your etl tasks

Indeed. This makes Prefect a great tool to combine the data science folk putting out Python model generators while still allowing the top of the funnel to be whatever the dev team needs.