Great question — there’s a small but important distinction here. - Port 4201 is the default for Prefect client metrics (e.g., workers/clients) — not the server. - For the Prefect 3 server, when you set

PREFECT_SERVER_METRICS_ENABLED=true

, the API serves Prometheus metrics on the same API port (4200 by default) at the path

/api/metrics

. So you do not need to add a new containerPort or a new Service for 4201. Just have Prometheus scrape the existing server Service on port 4200 at

/api/metrics

. How to proceed 1) Verify the endpoint - Port-forward and curl:

kubectl port-forward svc/<your-release-name>-server 4200:4200
curl <http://localhost:4200/api/metrics>

You should see Prometheus-formatted metrics. 2) Add scrape config via annotations (simple Prometheus) Add these to your Helm values for the server Service:

server:
  env:
    - name: PREFECT_SERVER_METRICS_ENABLED
      value: "true"

service:
  annotations:
    <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    <http://prometheus.io/port|prometheus.io/port>: "4200"
    <http://prometheus.io/path|prometheus.io/path>: "/api/metrics"

3) Or define a ServiceMonitor (Prometheus Operator) The chart’s Service port name is typically

server-svc-port

. A minimal ServiceMonitor looks like:

apiVersion: <http://monitoring.coreos.com/v1|monitoring.coreos.com/v1>
kind: ServiceMonitor
metadata:
  name: prefect-server
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      <http://app.kubernetes.io/name|app.kubernetes.io/name>: prefect-server
      <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: <your-release-name>
  endpoints:
    - port: server-svc-port
      path: /api/metrics
      scheme: http

Useful references - Helm chart (service template): https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/templates/service.yaml - Helm chart values: https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/values.yaml Optional: client/worker metrics If you later want metrics from Prefect workers/clients, those use port 4201 by default and would require exposing that port on those pods separately. That’s unrelated to the server metrics above. If you share your Helm values snippet, I can suggest the exact additions for your setup.

Got it — if you’re hitting /api/metrics and “get nothing,” there are a few likely culprits. Let’s quickly narrow it down. What to expect - Endpoint: /api/metrics on the same port as the API (default 4200) - If enabled and reachable: 200 OK with Prometheus text (lots of lines like python_info, process_cpu_seconds_total) - If not enabled: 404 Not Found - If API auth is enabled: 401 Unauthorized (metrics are not exempt from auth) - If ingress rewrites the path: you may need a different external path (e.g., /prefect/api/metrics) Quick checks 1) Confirm the setting is actually applied in the running pod

kubectl exec -it <prefect-server-pod> -- env | grep -E 'PREFECT_(SERVER_|API_).*METRICS'

You should see either: - PREFECT_SERVER_METRICS_ENABLED=true - or PREFECT_API_ENABLE_METRICS=true If not present, the env var wasn’t applied to the server container. In the Helm chart, set it under

server.env

(not just a global):

server:
  env:
    - name: PREFECT_SERVER_METRICS_ENABLED
      value: "true"

Chart values reference: https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/values.yaml 2) Bypass ingress and test directly

kubectl port-forward svc/<your-release-name>-server 4200:4200
curl -i <http://localhost:4200/api/metrics>

- 200 + text = good - 404 = setting not applied; redeploy after fixing env - 401 = basic auth is enabled; use:

curl -i -u <user>:<pass> <http://localhost:4200/api/metrics>

3) If you use an ingress with a path prefix If your UI/API are behind a prefix (e.g., /prefect), the external path becomes: - https://<host>/prefect/api/metrics Confirm your ingress isn’t stripping “/api” before it reaches the backend. As a sanity check:

curl -i https://<host>/.../api/openapi.json

If that 404s externally but works via port-forward, your ingress is rewriting paths; adjust the metrics path you scrape or update the rewrite rules. Prometheus scrape examples - Simple annotations on the Service (no auth):

service:
  annotations:
    <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    <http://prometheus.io/port|prometheus.io/port>: "4200"
    <http://prometheus.io/path|prometheus.io/path>: "/api/metrics"

Service template for reference: https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/templates/service.yaml - Prometheus Operator ServiceMonitor:

apiVersion: <http://monitoring.coreos.com/v1|monitoring.coreos.com/v1>
kind: ServiceMonitor
metadata:
  name: prefect-server
spec:
  selector:
    matchLabels:
      <http://app.kubernetes.io/name|app.kubernetes.io/name>: prefect-server
      <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: <your-release-name>
  endpoints:
    - port: server-svc-port
      path: /api/metrics
      scheme: http

If you’ve enabled basic auth on the server, add basic_auth in your Prometheus scrape config. If you can share: - The HTTP status/headers from

curl -i

- Whether API basic auth is enabled - Any ingress path prefix you’re using - The

server.env

section of your Helm values …I can pinpoint the exact fix. Links for reference: - Helm values.yaml: https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/values.yaml - Deployment template (env + ports): https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/templates/deployment.yaml - Prefect server metrics route (source): https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py