Thread
#prefect-server
    j

    Julian

    2 years ago
    Hey guys, we are currently evaluating prefect as a possibility to manage some of our workflows and it looks very promising. 🙂 We use prometheus to monitor all our services, so I want to monitor our flows as well. These run asynchroniously scheduled on local agents with dask executors and stored to s3 (as well as the flow results). Prometheus has a python_client https://github.com/prometheus/client_python, but I can't manage to integrate these metrics to our flows. I tried to include them in a task (say prometheus_monitor_task) as well as on callback, when a flow changes its state, but then I get the error, that the flow can't be pickled. How would you monitor your flows in addition to custom loggers? Is there any support/will there be support in the near future?
    Zachary Hughes

    Zachary Hughes

    2 years ago
    Hi @Julian! I'm not super familiar with Prometheus (reading up a bit now), but does this seem like something logging handlers might be able to help you with? https://docs.prefect.io/core/advanced_tutorials/custom-logs.html#adding-handlers
    j

    Julian

    2 years ago
    Thank you for your response. A post request to an external webserver might do the job. To follow up my question. How would you retrieve stats from flow runs/tasks, which can be seen in the ui, besides grepping the logs?
    Dylan

    Dylan

    2 years ago
    @Julian for high-level stats, you might want to try a State Handler: https://docs.prefect.io/core/concepts/notifications.html#state-handlers
    If you wrote one for Prometheus, it would ping on every state change. If you then added these to your flows/tasks, prometheus would get a high-level look at the states in your prefect flows
    j

    Julian

    2 years ago
    Thank you!
    Dylan

    Dylan

    2 years ago
    Of course!
    Since this is probably a common use case, we’d love to add your state handler to the Prefect repo
    Feel free to PR it!