Roman Borysenko
07/31/2025, 3:34 PMcurl: (52) Empty reply from server or {"detail":"Not Found"}. Is there any documentation on how to do it?Brendan Dalpe
07/31/2025, 5:16 PMPREFECT_CLIENT_ENABLE_METRICS environment variable.
These links might be useful:
⢠https://linen.prefect.io/t/23042222/hey-folks-i-quite-like-the-prometheus-exporter-but-i-want-to
⢠https://linen.prefect.io/t/29208700/ulva73b9p-oss-self-hosted-prefect-on-k8s-do-i-need-to-set-th
⢠https://docs.prefect.io/v3/api-ref/settings-ref#clientmetricssettingsRoman Borysenko
08/01/2025, 5:38 AMPREFECT_CLIENT_ENABLE_METRICS and PREFECT_CLIENT_METRICS_PORT, but metrics server has not started, also no error. I am going to look into the source code of WorkerRoman Borysenko
08/01/2025, 5:53 AMkubectl exec -it -n prefect "$(kubectl get pods -n prefect -l <http://app.kubernetes.io/instance=prefect-sandbox-worker|app.kubernetes.io/instance=prefect-sandbox-worker> --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')" -- env | grep METRICS
Defaulted container "prefect-worker" out of: prefect-worker, sync-base-job-template (init)
PREFECT_CLIENT_ENABLE_METRICS=true
PREFECT_CLIENT_METRICS_PORT=4201
curl <http://localhost:4201/metrics>
curl: (52) Empty reply from server
E0801 08:49:26.007197 271707 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 4201 -> 4201: error forwarding port 4201 to pod 4c03b0a6e4c6f9a81b04388a8ec64f19f9044508fad5fe12ae8588025d43ddf6, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-8468ce59-edc8-34b1-bf96-ee63d729eea6\": failed to connect to localhost:4201 inside namespace \"4c03b0a6e4c6f9a81b04388a8ec64f19f9044508fad5fe12ae8588025d43ddf6\", IPv4: dial tcp4 127.0.0.1:4201: connect: connection refused IPv6 dial tcp6 [::1]:4201: connect: connection refused "
error: lost connection to podRoman Borysenko
08/01/2025, 1:33 PMversion: '3.8'
services:
prefect-server:
image: prefecthq/prefect:3.4-python3.12-kubernetes
command:
- /usr/bin/tini
- -g
- -s
- --
- /opt/prefect/entrypoint.sh
- prefect
- server
- start
ports:
- "4200:4200"
restart: always
environment:
PREFECT_SERVER_API_HOST: 0.0.0.0
PREFECT_SERVER_TELEMETRY_ENABLED: "false"
PREFECT_SERVER_METRICS_ENABLED: "true"
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; exit(0) if urllib.request.urlopen('<http://localhost:4200/api/health').status==200> else exit(1)"]
interval: 10s
timeout: 5s
retries: 30
prefect-worker:
image: prefecthq/prefect:3.4-python3.12-kubernetes
command: prefect worker start --pool "my-local-pool" --type "process"
ports:
- "4201:4201"
restart: on-failure
depends_on:
prefect-server:
condition: service_healthy
environment:
PREFECT_API_URL: <http://prefect-server:4200/api>
PREFECT_CLIENT_METRICS_ENABLED: "true"
PREFECT_CLIENT_METRICS_PORT: "4201"
Then:
docker-compose up
# wait till server and worker started and:
curl <http://localhost:4201/metrics>
curl: (56) Recv failure: Connection reset by peerBrendan Dalpe
08/01/2025, 3:30 PMBrendan Dalpe
08/01/2025, 3:46 PMRoman Borysenko
08/01/2025, 3:55 PMRoman Borysenko
08/01/2025, 3:55 PMcurl -s <http://localhost:8080/metrics> | grep ^prefect_
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/concurrency_limits/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/concurrency_limits/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/deployments/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/deployments/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/events/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/events/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/flow_runs/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/flow_runs/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/task_runs/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/task_runs/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/work_pools/filter>",method="POST",status=""} 29
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/work_pools/filter>",method="POST",status="200"} 29
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/work_pools/my-local-pool/workers/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/work_pools/my-local-pool/workers/filter>",method="POST",status="200"} 28
prefect_client_flow_run_failures_total{work_pool_name="my-local-pool"} 0
prefect_client_task_run_failures_total{work_pool_name="my-local-pool"} 0
prefect_deployments_total{work_pool_name="my-local-pool"} 0
prefect_worker_offline{work_pool_name="my-local-pool",worker_name="ProcessWorker 4e0b4633-044b-440c-b342-d84d6494bc5f"} 0
prefect_workers_count{status="HALTED",work_pool_name="my-local-pool"} 0
prefect_workers_count{status="OFFLINE",work_pool_name="my-local-pool"} 0
prefect_workers_count{status="ONLINE",work_pool_name="my-local-pool"} 1
prefect_workers_count{status="UNKNOWN",work_pool_name="my-local-pool"} 0Roman Borysenko
08/01/2025, 3:55 PMBrendan Dalpe
08/01/2025, 3:56 PMBrendan Dalpe
08/01/2025, 3:56 PMRoman Borysenko
08/01/2025, 3:58 PMRoman Borysenko
08/01/2025, 3:58 PMBrendan Dalpe
08/01/2025, 3:59 PMRoman Borysenko
08/01/2025, 3:59 PMBrendan Dalpe
08/01/2025, 3:59 PMRoman Borysenko
08/01/2025, 3:59 PMBrendan Dalpe
08/01/2025, 4:00 PMBrendan Dalpe
08/01/2025, 4:00 PM