How should web applications (Node.js) listen/subscribe to Prefect workflows completion? I have a task that trains a deep learning model through a Web Application. The “*Start training*” button on the web app sends a

POST /model

REST API call to the Node.js server which then in turn calls Prefect 2.0 using the REST API to start a new run (of an existing deployment). The workflow will train the model, run inference on some data and then persist the output to a database. This is obviously an asynchronous operation that may take a few minutes (or more) to complete. Assuming the workflow succeeded, I would like to notify the users (those who have Chrome opened to my web app) that something has happened, i.e. training completed. How should the Node.js be notified when the flow has finished (either successfully / failed)? Is there like a middleware / RabbitMQ / other message queue that the Node.js app can subscribe to, onto which Prefect publishes event? If not, does Prefect expose other broadcast events? And if not, should I poll periodically from my app and maintain state diff? Thanks!

You can add a callback method to the on_completion argument for a flow https://docs.prefect.io/api-ref/prefect/flows/

Thanks @Austin Weisgrau, the callback functions are only within the Prefect workflow.

Your callback function can issue an HTTP request to a webhook/endpoint on the node server. Simple example:

import requests
def notify_on_completion(flow, flow_run, state):
    <http://requests.post|requests.post>(webhook_url, json={'flow_name': flow.name, 'result': state.result()})

@flow(on_completion=[notify_on_completion])
def my_flow():
   ...

Thanks! Webhook is the modern PubSub way to communicate between different distributed services? I know that RabbitMQ (ampq clients) and Kafka is also an option

Ah gotcha! I think that choice depends on more of the specifications of the project. Kafka is good for really high volume queues with a lot of configurable control over how to handle outages, etc. A webhook is easy to implement if the situation is lower volume or not too sophisticated. IDK about RabbitMQ personally

Easy is good, our requirements are pretty lean