Hey guys, we are currently evaluating prefect as a...
# prefect-server
Hey guys, we are currently evaluating prefect as a possibility to manage some of our workflows and it looks very promising. 🙂 We use prometheus to monitor all our services, so I want to monitor our flows as well. These run asynchroniously scheduled on local agents with dask executors and stored to s3 (as well as the flow results). Prometheus has a python_client https://github.com/prometheus/client_python, but I can't manage to integrate these metrics to our flows. I tried to include them in a task (say prometheus_monitor_task) as well as on callback, when a flow changes its state, but then I get the error, that the flow can't be pickled. How would you monitor your flows in addition to custom loggers? Is there any support/will there be support in the near future?
Hi @Julian! I'm not super familiar with Prometheus (reading up a bit now), but does this seem like something logging handlers might be able to help you with? https://docs.prefect.io/core/advanced_tutorials/custom-logs.html#adding-handlers
Thank you for your response. A post request to an external webserver might do the job. To follow up my question. How would you retrieve stats from flow runs/tasks, which can be seen in the ui, besides grepping the logs?
@Julian for high-level stats, you might want to try a State Handler: https://docs.prefect.io/core/concepts/notifications.html#state-handlers
upvote 1
If you wrote one for Prometheus, it would ping on every state change. If you then added these to your flows/tasks, prometheus would get a high-level look at the states in your prefect flows
Thank you!
Of course!
Since this is probably a common use case, we’d love to add your state handler to the Prefect repo
Feel free to PR it!