Great question — and you’re on the right track. Assuming you’re on Prefect 3.x: - There is no

prefect server metrics

CLI command in 3.x (that subcommand does not exist). - The official way to expose metrics is: - A Prometheus

/api/metrics

endpoint on the Prefect Server - A lightweight Prometheus HTTP server in each Prefect client/worker process Below are the official ways to enable both, with docs and code. How to expose Prometheus metrics from Prefect Server - Enable the server’s Prometheus endpoint and restart your server:

# via environment variable
export PREFECT_SERVER_METRICS_ENABLED=true
# legacy alias also accepted: PREFECT_API_ENABLE_METRICS=true

# or persist to your profile
prefect config set PREFECT_SERVER_METRICS_ENABLED=true

# start the server
prefect server start

- Verify:

curl <http://localhost:8080/api/metrics>

- Docs and source - Settings reference: Prefect settings reference - Server API settings: Server API settings (docs) - Source (settings): prefect/settings/models/server/api.py - Source (endpoint): prefect/server/api/server.py (exposes /api/metrics) How to expose per-worker Prometheus metrics Enable the client metrics HTTP server in each worker process (this gives you per-worker metrics to scrape):

# set on each worker
export PREFECT_CLIENT_METRICS_ENABLED=true
export PREFECT_CLIENT_METRICS_PORT=4201  # default is 4201

# then start the worker
prefect worker start -p <your-work-pool>

- Verify:

curl http://<worker-host>:4201/metrics

- Docs and source - Client metrics settings: Client metrics settings (docs) - Settings reference: Prefect settings reference - Source (settings): prefect/settings/models/client.py - Source (service): start_client_metrics_server() docs What you can monitor per worker - From the API/UI: worker ONLINE/OFFLINE status, last heartbeat, and heartbeat interval. - CLI:

prefect work-pool inspect "my-pool" --output json

- UI: Work Pools page shows worker list, status, and last heartbeat. - From Prometheus (per worker): client-side Prefect metrics (events/log websocket counters, etc.) on the port you enable (e.g., 4201). - Note: There is no built-in “active runs per worker” metric exposed by the API today. Workers do track concurrency internally, but that count is not exported as a Prometheus metric. To correlate load, combine: - Worker heartbeats + status (UI/API) - Flow run states by pool/queue (API) - Infra-level metrics (CPU/memory) via node-exporter/cAdvisor or OTLP resource metrics (see below) Optional: OS-level resource metrics (OpenTelemetry) If you want CPU/memory usage for flow run subprocesses exported to an OTLP collector:

export PREFECT_TELEMETRY_ENABLE_RESOURCE_METRICS=true
# Point to your OTLP collector
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="<http://collector:4318/v1/metrics>"
# or
export OTEL_EXPORTER_OTLP_ENDPOINT="<http://collector:4318>"

- Source (resolution logic): prefect/telemetry/_metrics.py - Settings (telemetry): prefect/settings/models/telemetry.py Prometheus scrape examples - Prometheus config to scrape both the server and a worker: ``` global: scrape_interval: 15s scrape_configs: - job_name: 'prefect-server'

static_configs: - targets: ['server-host:8080'] metrics_path: '/api/metrics' - job_name: 'prefect-worker' static_configs: - targets: ['worker-host:4201']

Minimal Docker Compose example

services: prefect-server: image: prefecthq/prefect:latest environment: PREFECT_SERVER_METRICS_ENABLED: "true" PREFECT_API_URL: "http://0.0.0.0:8080/api" ports: - "8080:8080" command: prefect server start worker-a: image: prefecthq/prefect:latest environment: PREFECT_API_URL: "http://prefect-server:8080/api" PREFECT_CLIENT_METRICS_ENABLED: "true" PREFECT_CLIENT_METRICS_PORT: "4201" ports: - "4201:4201" command: prefect worker start -p default depends_on: - prefect-server ``` A couple of important notes - There is no

prefect server metrics

CLI command in Prefect 3.x. Use the

/api/metrics

endpoint by enabling server metrics as shown above. - Per-worker “load” is best approximated today by: - Worker status and heartbeats (UI/CLI/API) - Scraping each worker’s client metrics port - Infra-level metrics (CPU/memory) via exporters or OTLP - Optionally, query running flow runs by work pool/queues to see demand If you share how you’re running workers (Docker, Kubernetes, systemd), I can provide a tailored snippet for that environment.

Thanks for sharing the output — that helps. What you’re seeing on /api/metrics after setting PREFECT_SERVER_METRICS_ENABLED=true are the standard Prometheus Python client process/runtime metrics plus a few Prefect-specific counters. Quick guide: - python_gc_objects_collected_total{generation="0|1|2"}: Total objects collected by Python’s GC per generation since this process started. - python_gc_objects_uncollectable_total{generation="…"}: Uncollectable (leaked) objects found by GC. Usually 0. - python_gc_collections_total{generation="…"}: How many GC cycles have run per generation. - python_info{implementation,major,minor,patchlevel,version}: Static 1.0 gauge with Python version info. - process_virtual_memory_bytes: Total virtual memory size of the process. - process_resident_memory_bytes: Resident set size (actual RAM used). - process_start_time_seconds: When this process started (epoch seconds). - process_cpu_seconds_total: Total CPU time this process has consumed. - process_open_fds / process_max_fds: Current and max file descriptors in this process. Prefect-specific counters (emitted by the server process): - prefect_events_emitted_total: Number of Prefect events emitted by this process. - prefect_events_observed_total: Events this process observed/subscribed to. - prefect_event_websocket_connections_total{direction,connection}: Event websocket connection attempts (direction “in/out”, connection “initial/reconnect”). - prefect_event_websocket_checkpoints_total: Event stream checkpoints performed. - prefect_logs_observed_total: Log messages observed by this process’s log subscriber(s). - prefect_log_websocket_connections_total{direction,connection}: Log websocket connection attempts. About “per server worker” metrics with --workers 4 - prefect server start --workers 4 starts multiple uvicorn worker processes. - Prefect’s /api/metrics endpoint uses the default, in-process Prometheus registry (no multiprocess aggregator). - That means each scrape returns metrics for the single uvicorn worker that happened to handle that HTTP request. There’s no worker label and no built-in way to pick a specific worker. If you curl repeatedly, you may hit different workers and see different values. - Prefect Server does not currently wire up prometheus_client’s multiprocess collector (PROMETHEUS_MULTIPROC_DIR + MultiProcessCollector). Even if it did, multiprocess mode aggregates across workers into one output, not separate per-worker series. Options if you need per-worker visibility - Run 1 worker per API replica and scale horizontally: - Start each Prefect Server instance with --workers 1 and run multiple replicas (e.g., multiple containers/pods). - Have Prometheus scrape each replica individually; you’ll effectively get per-replica (per-worker) metrics via the target label. - If you need true per-uvicorn-worker metrics in a single process group, that’s not supported out of the box. You’d need custom code to: - Add a metrics endpoint that uses the multiprocess collector (aggregated across workers), or - Expose per-worker endpoints/labels (non-trivial with uvicorn’s worker model). Official references - Settings reference (server and client metrics): Prefect settings reference - Server API settings (metrics toggle): Server API settings (docs) - Source of /api/metrics implementation: prefect/server/api/server.py If your goal is to understand API load distribution across the four uvicorn workers, the best supported pattern today is to run a single worker per replica and scrape each replica separately. If you were instead asking about Prefect “execution workers” (the processes polling work pools), let me know — that’s a different surface (and the per-process Prometheus server there is enabled with PREFECT_CLIENT_METRICS_ENABLED).

✅ thought for 93.8 seconds

That sounds frustrating. Kernel slab growth points to resources the kernel is holding onto (file caches, dentries/inodes, sockets, etc.), not typical Python heap leaks. Prefect won’t directly allocate kernel slab, but a long‑running worker can drive the kernel to grow these caches over time. To help pinpoint and mitigate, here’s a focused plan. First, a bit of triage: what’s growing? - Capture slab details over time

slabtop -sc   # live view, sort by cache size
  cat /proc/meminfo | egrep 'Slab|SReclaimable|SUnreclaim'
  cat /proc/slabinfo | sort -k3 -n | tail -n 25
  egrep 'dentry|inode_cache|kmalloc|skbuff' /proc/slabinfo | sort -k3 -n | tail -n 10

- dentry/inode_cache growth → high file churn (lots of files/dirs created/read). - skbuff_* growth → high/long‑lived TCP traffic or many idle sockets. - kmalloc-* growth → generic kernel allocations; correlate with other signals. - Check for file descriptor/socket leakage by the worker process

PID=$(pgrep -f 'prefect worker start')
  ls -l /proc/$PID/fd | wc -l
  lsof -p $PID | wc -l
  ss -s
  ss -tanp | grep $PID

- Check temp/work directories for churn or orphaned files

du -sh /tmp
  find /tmp -maxdepth 1 -type d -name 'prefect*' -print

Environment questions (to tailor next steps) - How are you running the worker: Process, Docker, or Kubernetes worker? - OS/distro and kernel version; container runtime (Docker/Moby/containerd), if any. - Prefect version and Python version. - Approx. flow runs/day and whether they create many small files or large logs locally. - Are you persisting results/logs locally or to remote storage? - Any resource limits on the worker container/host? Common causes and Prefect-aware mitigations - Heavy file churn → dentry/inode slab growth - Ensure flow runs are not leaving temp dirs/files behind (e.g., if runs crash before cleanup). - Periodically prune temp dirs:

sudo find /tmp -maxdepth 1 -type d -name 'prefect*' -mtime +1 -exec rm -rf {} +

- Prefer remote result storage (object storage) over local disk for large/temporary outputs. - Many open/idle sockets → skbuff_* growth - Look for reconnect storms (network instability to API). The worker should have only a small number of steady connections. - If you see a large/monotonic count of TCP connections for the worker PID, we should investigate logs/events clients or any custom code opening sockets and not closing. - Docker hosts (Process/Docker workers) - Configure container log rotation (json‑file driver) to avoid inode/dentry churn:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

- Prune unused images/volumes regularly (carefully):

docker system prune

and

docker image prune

. - Kubernetes nodes - Ensure container log rotation is enabled on the node. - Check if pods mount paths with high file churn; verify that cleanup jobs run. - Reduce worker concurrency → less churn per host - Set a work pool concurrency limit to cap simultaneous runs:

prefect work-pool set-concurrency-limit "my-pool"  N

Then observe slab trends again. - Make restarts intentional - If a slow kernel-memory creep is unavoidable in your environment or workload, schedule a controlled restart of the worker (e.g., daily) via your process manager or orchestrator rather than waiting for OOM. Add observability for correlation - Enable per‑worker Prometheus metrics (Prefect client metrics):

export PREFECT_CLIENT_METRICS_ENABLED=true
  export PREFECT_CLIENT_METRICS_PORT=4201
  prefect worker start -p <your-pool>

Then scrape

http://<worker-host>:4201/metrics

alongside node‑level exporters (e.g., node_exporter) so you can correlate flow activity with slab growth. - Client metrics setting ref: Client metrics settings (docs)

- Settings ref: All Prefect settings What to share next so we can pinpoint root cause - Top 10 lines from

slabtop -sc

(after a few days of uptime). - Trend of

ls -l /proc/$PID/fd | wc -l

for the worker process over time. - Whether you’re on bare metal, Docker, or Kubernetes (plus versions). - Any evidence of leftover temp dirs or large local log/result directories. With those, I can recommend targeted fixes (e.g., cleaning a specific temp path, adjusting logging, or addressing connection churn).