https://prefect.io logo
Title
o

Ofir

03/02/2023, 11:47 PM
How should web applications (Node.js) listen/subscribe to Prefect workflows completion? I have a task that trains a deep learning model through a Web Application. The “*Start training*” button on the web app sends a
POST /model
REST API call to the Node.js server which then in turn calls Prefect 2.0 using the REST API to start a new run (of an existing deployment). The workflow will train the model, run inference on some data and then persist the output to a database. This is obviously an asynchronous operation that may take a few minutes (or more) to complete. Assuming the workflow succeeded, I would like to notify the users (those who have Chrome opened to my web app) that something has happened, i.e. training completed. How should the Node.js be notified when the flow has finished (either successfully / failed)? Is there like a middleware / RabbitMQ / other message queue that the Node.js app can subscribe to, onto which Prefect publishes event? If not, does Prefect expose other broadcast events? And if not, should I poll periodically from my app and maintain state diff? Thanks!
a

Austin Weisgrau

03/03/2023, 1:10 AM
You can add a callback method to the on_completion argument for a flow https://docs.prefect.io/api-ref/prefect/flows/
🙌 1
o

Ofir

03/03/2023, 2:23 PM
Thanks @Austin Weisgrau, the callback functions are only within the Prefect workflow.
My question is how does an external agent decoupled from Prefect should know that it finished?
Let’s say a web server (e.g. Node.js) that wants to be notified
a

Austin Weisgrau

03/03/2023, 5:50 PM
Your callback function can issue an HTTP request to a webhook/endpoint on the node server. Simple example:
import requests
def notify_on_completion(flow, flow_run, state):
    <http://requests.post|requests.post>(webhook_url, json={'flow_name': flow.name, 'result': state.result()})

@flow(on_completion=[notify_on_completion])
def my_flow():
   ...
o

Ofir

03/03/2023, 5:52 PM
Thanks! Webhook is the modern PubSub way to communicate between different distributed services? I know that RabbitMQ (ampq clients) and Kafka is also an option
but don’t have experience with either
a

Austin Weisgrau

03/03/2023, 5:53 PM
Ah gotcha! I think that choice depends on more of the specifications of the project. Kafka is good for really high volume queues with a lot of configurable control over how to handle outages, etc. A webhook is easy to implement if the situation is lower volume or not too sophisticated. IDK about RabbitMQ personally
o

Ofir

03/03/2023, 5:54 PM
Easy is good, our requirements are pretty lean
Getting notified on completion is a very good start. We’ll reiterate as we need more changes.
Is there a Prefect + Flask/Django or Prefect + Node.js complete example where I can see how the setup of the webhook works, and how the overall scheme (connect and listen to the webhook) is orchestrated?
BTW, doesn’t it create a tight coupling between my backend server and the Prefect orchestrator?
AFAICT it’s better if the Webhook HTTP endpoint is decoupled from both and serves merely as a billboard / broadcast channel. In terms of deployment and container orchestration it means: 1. Node.js Docker container 2. Separate Web server with one Webhook HTTP endpoint 3. Prefect Dockers