Hey guys, we are currently evaluating prefect as a possibility to manage some of our workflows and it looks very promising. 🙂
We use prometheus to monitor all our services, so I want to monitor our flows as well. These run asynchroniously scheduled on local agents with dask executors and stored to s3 (as well as the flow results). Prometheus has a python_client https://github.com/prometheus/client_python, but I can't manage to integrate these metrics to our flows. I tried to include them in a task (say prometheus_monitor_task) as well as on callback, when a flow changes its state, but then I get the error, that the flow can't be pickled.
How would you monitor your flows in addition to custom loggers? Is there any support/will there be support in the near future?
Thank you for your response. A post request to an external webserver might do the job.
To follow up my question. How would you retrieve stats from flow runs/tasks, which can be seen in the ui, besides grepping the logs?
If you wrote one for Prometheus, it would ping on every state change. If you then added these to your flows/tasks, prometheus would get a high-level look at the states in your prefect flows
j
Julian
07/07/2020, 3:43 PM
Thank you!
d
Dylan
07/07/2020, 3:45 PM
Of course!
Since this is probably a common use case, we’d love to add your state handler to the Prefect repo