How should web applications (Node.js) listen/subscribe to Prefect workflows completion? I have a task that trains a deep learning model through a Web Application. The “*Start training*” button on the web app sends a

POST /model

REST API call to the Node.js server which then in turn calls Prefect 2.0 using the REST API to start a new run (of an existing deployment). The workflow will train the model, run inference on some data and then persist the output to a database. This is obviously an asynchronous operation that may take a few minutes (or more) to complete. Assuming the workflow succeeded, I would like to notify the users (those who have Chrome opened to my web app) that something has happened, i.e. training completed. How should the Node.js be notified when the flow has finished (either successfully / failed)? Is there like a middleware / RabbitMQ / other message queue that the Node.js app can subscribe to, onto which Prefect publishes event? If not, does Prefect expose other broadcast events? And if not, should I poll periodically from my app and maintain state diff? Thanks!

Hey Ofir, thanks for your question, I’m not a front end person so I’m also going to tag in my colleague @Craig Harshbarger for a perspective. What I know would work is if you set up a webhook url to listen to, you could configure automations in prefect cloud to fire events off to your webhook url, but I have a sense modern webapps may be slightly more sophisticated than that. We do not have an outgoing state event queue to subscribe to at this time.

You could definitely set up an Express.js server and receive web hook calls from Prefect. Or NestJS, or Fastify instead of Express, if you want. Any Node web framework that lets you set up an HTTP(S) endpoint should work fine. As for your users: maybe use websocket connections to send them push notifications when the thing they care about finishes? Although if you're using Node, it might be easier to use socket.io instead of raw websockets to get seamless fallback to long polling and a friendlier API.

Thanks Ryan this is brilliant! Doesn’t it create a loop / tight coupling between the consumer (web app) and the producer (prefect orchestrator)? What’s the best practice for this integration scenario, spawn a new web app that will expose just one Websocket endpoint and have Prefect publish to that and the other Node server to subscribe to that?

What you suggest could work if you want to decouple things. The app that Prefect publishes to would need an HTTP endpoint to receive Prefect web hook calls. You could connect the two Node apps together via a web socket, but it might be more resilient and scalable if you connect them using something like RabbitMQ. I could see Redis pub/sub working as a connector too, depending on how you want to to build things. I guess at that point it's kind of a distributed systems design question, and the best answer depends on your specific needs.

Thanks 🙏 I’ll follow the yc approach of “write code that doesn’t scale” and I’ll KISS

Agreed 🙂 scaling is a nice problem to have, but good to start simple and scale when you need to