Hello! I am trying to enable Prefect Worker (K8s) ...
# ask-community
r
Hello! I am trying to enable Prefect Worker (K8s) metrics for Prometheus, but having issues:
curl: (52) Empty reply from server
or
{"detail":"Not Found"}
. Is there any documentation on how to do it?
b
r
@Brendan Dalpe Thank you! I tried
PREFECT_CLIENT_ENABLE_METRICS
and
PREFECT_CLIENT_METRICS_PORT
, but metrics server has not started, also no error. I am going to look into the source code of Worker
Copy code
kubectl exec -it -n prefect "$(kubectl get pods -n prefect -l <http://app.kubernetes.io/instance=prefect-sandbox-worker|app.kubernetes.io/instance=prefect-sandbox-worker> --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')" -- env | grep METRICS
Defaulted container "prefect-worker" out of: prefect-worker, sync-base-job-template (init)
PREFECT_CLIENT_ENABLE_METRICS=true
PREFECT_CLIENT_METRICS_PORT=4201

curl <http://localhost:4201/metrics>
curl: (52) Empty reply from server

E0801 08:49:26.007197  271707 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 4201 -> 4201: error forwarding port 4201 to pod 4c03b0a6e4c6f9a81b04388a8ec64f19f9044508fad5fe12ae8588025d43ddf6, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-8468ce59-edc8-34b1-bf96-ee63d729eea6\": failed to connect to localhost:4201 inside namespace \"4c03b0a6e4c6f9a81b04388a8ec64f19f9044508fad5fe12ae8588025d43ddf6\", IPv4: dial tcp4 127.0.0.1:4201: connect: connection refused IPv6 dial tcp6 [::1]:4201: connect: connection refused "
error: lost connection to pod
Here is example to reproduce the case:
Copy code
version: '3.8'

services:
  prefect-server:
    image: prefecthq/prefect:3.4-python3.12-kubernetes
    command:
      - /usr/bin/tini
      - -g
      - -s
      - --
      - /opt/prefect/entrypoint.sh
      - prefect
      - server
      - start
    ports:
      - "4200:4200"
    restart: always
    environment:
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_SERVER_TELEMETRY_ENABLED: "false"
      PREFECT_SERVER_METRICS_ENABLED: "true"
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; exit(0) if urllib.request.urlopen('<http://localhost:4200/api/health').status==200> else exit(1)"]
      interval: 10s
      timeout: 5s
      retries: 30

  prefect-worker:
    image: prefecthq/prefect:3.4-python3.12-kubernetes
    command: prefect worker start --pool "my-local-pool" --type "process"
    ports:
      - "4201:4201"
    restart: on-failure
    depends_on:
      prefect-server:
        condition: service_healthy
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
      PREFECT_CLIENT_METRICS_ENABLED: "true"
      PREFECT_CLIENT_METRICS_PORT: "4201"
Then:
Copy code
docker-compose up
# wait till server and worker started and:
curl <http://localhost:4201/metrics>
curl: (56) Recv failure: Connection reset by peer
b
Did some digging into the code and it looks like metrics isn't started for workers, only runners šŸ˜ž
r
I have created custom metrics exporter
āœ… 1
Copy code
curl -s <http://localhost:8080/metrics> | grep ^prefect_
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/concurrency_limits/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/concurrency_limits/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/deployments/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/deployments/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/events/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/events/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/flow_runs/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/flow_runs/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/task_runs/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/task_runs/filter>",method="POST",status="200"} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/work_pools/filter>",method="POST",status=""} 29
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/work_pools/filter>",method="POST",status="200"} 29
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/work_pools/my-local-pool/workers/filter>",method="POST",status=""} 28
prefect_client_api_requests_total{endpoint="<http://prefect-server:4200/api/work_pools/my-local-pool/workers/filter>",method="POST",status="200"} 28
prefect_client_flow_run_failures_total{work_pool_name="my-local-pool"} 0
prefect_client_task_run_failures_total{work_pool_name="my-local-pool"} 0
prefect_deployments_total{work_pool_name="my-local-pool"} 0
prefect_worker_offline{work_pool_name="my-local-pool",worker_name="ProcessWorker 4e0b4633-044b-440c-b342-d84d6494bc5f"} 0
prefect_workers_count{status="HALTED",work_pool_name="my-local-pool"} 0
prefect_workers_count{status="OFFLINE",work_pool_name="my-local-pool"} 0
prefect_workers_count{status="ONLINE",work_pool_name="my-local-pool"} 1
prefect_workers_count{status="UNKNOWN",work_pool_name="my-local-pool"} 0
maybe will need to adjust it or add more
b
So you're scraping the Prefect Server to get the desired metrics?
Would give better stats than what we export currently
r
yes
I use Go lang
b
r
we are using it
šŸ™Œ 1
b
Let us know if there's some functionality missing
r
thanks
b
Ah, I see your issue now