<@ULVA73B9P> Does Prefect self hosted 3.6.12 suppo...
# ask-marvin
n
@Marvin Does Prefect self hosted 3.6.12 support Kubernetes metrics? And what metrics does it expose?
m
thought for 152.9 seconds
Yes — Prefect self-hosted 3.6.12 supports metrics export, but not Kubernetes-native metrics. What it supports - A Prometheus/OpenMetrics endpoint on the Prefect server API - A separate client/worker metrics endpoint for workers and other Prefect client processes What it does not support - No built-in Kubernetes-specific metrics like pod status, node usage, scheduler state, or kube API object metrics - For those, you’d still use standard Kubernetes observability tools like
kube-state-metrics
, node exporter, cAdvisor, etc. Server/API metrics - Enable with
PREFECT_API_ENABLE_METRICS=true
- Alias also exists:
PREFECT_SERVER_METRICS_ENABLED=true
- Endpoint:
/api/metrics
- Served in Prometheus/OpenMetrics text format So if your self-hosted API is at:
<http://prefect-server:4200>
You’d scrape:
<http://prefect-server:4200/api/metrics>
What the server exposes Prefect 3.6.12’s API metrics endpoint mainly exposes the default metrics provided by
prometheus_client
, including: -
process_*
metrics - CPU time - resident/virtual memory - file descriptors, etc. -
python_*
metrics - GC activity - interpreter/runtime stats - threads, etc. I did not find evidence of a rich set of Prefect-server-specific business metrics on that endpoint in 3.6.12. Worker/client metrics Workers can also expose metrics from a separate HTTP server. Enable with: -
PREFECT_CLIENT_METRICS_ENABLED=true
- Optional port override:
PREFECT_CLIENT_METRICS_PORT=4201
Default port: -
4201
Worker/client metrics exposed These Prefect-specific metrics are present: -
prefect_logs_observed
- Labels:
client
-
prefect_log_websocket_connections
- Labels:
client
,
direction
,
connection
-
prefect_events_emitted
- Labels:
client
-
prefect_events_observed
- Labels:
client
-
prefect_event_websocket_connections
- Labels:
client
,
direction
,
connection
-
prefect_event_websocket_checkpoints
- Labels:
client
Where label values include things like: - `direction`:
in
,
out
- `connection`:
initial
,
reconnect
In Kubernetes This means: - You can scrape Prefect server and worker pods with Prometheus - But the metrics are application/process metrics, not Kubernetes control-plane or pod lifecycle metrics Typical setup: - Scrape Prefect API at
/api/metrics
- Scrape worker pods on port
4201
- Use a
ServiceMonitor
or PodMonitor if you’re using Prometheus Operator Example env vars for the API pod:
Copy code
- name: PREFECT_API_ENABLE_METRICS
  value: "true"
Example env vars for a worker pod:
Copy code
- name: PREFECT_CLIENT_METRICS_ENABLED
  value: "true"
- name: PREFECT_CLIENT_METRICS_PORT
  value: "4201"
Bottom line - Yes: self-hosted Prefect 3.6.12 supports Prometheus metrics in Kubernetes - No: it does not expose Kubernetes-specific metrics out of the box - Exposed metrics: mostly
process_*
,
python_*
, plus a small set of Prefect client/worker event/log websocket counters If you want, I can also give you a ready-to-apply: - Kubernetes
Service
+
ServiceMonitor
for the API -
Service
+
ServiceMonitor
for workers - sample Prometheus scrape config