Great question. In Prefect 3.x, the most reliable way to get OpenTelemetry (OTel) traces/metrics/logs from both the Prefect Server (FastAPI/Uvicorn) and your Workers/flows is to run those processes under OpenTelemetry auto-instrumentation and point them at your OTLP destination (OTel Collector, Datadog, Tempo, etc.). Prefect does not ship a single “enable OTEL” switch for the server or workers; instead, you use standard OTel instrumentation and env vars. Below are practical recipes for local, Docker, and Kubernetes. If you can share where you’re running Server/Workers and which backend you want to send to (OTLP Collector vs Datadog vs Tempo), I’ll tailor these to your setup. What you’ll need - Install OTel tooling into the same environment/container as Prefect: - opentelemetry-distro (auto-instrumentation entrypoint) - opentelemetry-exporter-otlp - Optional: opentelemetry-instrumentation-fastapi, opentelemetry-instrumentation-uvicorn, and other libs your stack uses - Set OTLP exporter env vars in the environment where Prefect Server/Workers run: - OTEL_EXPORTER_OTLP_ENDPOINT (e.g., http://otel-collector:4317 for gRPC or http://otel-collector:4318 for HTTP) - OTEL_TRACES_EXPORTER=otlp - OTEL_METRICS_EXPORTER=otlp - OTEL_LOGS_EXPORTER=otlp (optional; logs support varies by backend) - OTEL_SERVICE_NAME=prefect-server or prefect-worker (or any service name you prefer) Instrument the Prefect Server - Local - Install OTel packages:

pip install "prefect>=3,<4" opentelemetry-distro opentelemetry-exporter-otlp
    opentelemetry-bootstrap -a install

- Run the Prefect server under the OTel wrapper:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
    OTEL_TRACES_EXPORTER=otlp \
    OTEL_METRICS_EXPORTER=otlp \
    OTEL_SERVICE_NAME=prefect-server \
    opentelemetry-instrument prefect server start --host 0.0.0.0

- Docker Compose - Example service definition:

services:
      prefect-server:
        image: prefecthq/prefect:3-latest
        command: >
          sh -c "pip install opentelemetry-distro opentelemetry-exporter-otlp &&
                 opentelemetry-bootstrap -a install &&
                 opentelemetry-instrument prefect server start --host 0.0.0.0 --ui"
        environment:
          - OTEL_EXPORTER_OTLP_ENDPOINT=<http://otel-collector:4317>
          - OTEL_TRACES_EXPORTER=otlp
          - OTEL_METRICS_EXPORTER=otlp
          - OTEL_LOGS_EXPORTER=otlp
          - OTEL_SERVICE_NAME=prefect-server
        depends_on:
          - otel-collector

- Kubernetes - Wrap the server container command with opentelemetry-instrument and add OTEL_* env vars:

containers:
    - name: prefect-server
      image: prefecthq/prefect:3-latest
      env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: <http://otel-collector:4317>
        - name: OTEL_TRACES_EXPORTER
          value: otlp
        - name: OTEL_METRICS_EXPORTER
          value: otlp
        - name: OTEL_SERVICE_NAME
          value: prefect-server
      command: ["sh", "-c"]
      args:
        - >
          pip install opentelemetry-distro opentelemetry-exporter-otlp &&
          opentelemetry-bootstrap -a install &&
          opentelemetry-instrument prefect server start --host 0.0.0.0 --ui

Instrument Prefect Workers (and your flows) - Local - Run your worker under OTel:

OTEL_EXPORTER_OTLP_ENDPOINT=<http://localhost:4317> \
    OTEL_TRACES_EXPORTER=otlp \
    OTEL_METRICS_EXPORTER=otlp \
    OTEL_SERVICE_NAME=prefect-worker \
    opentelemetry-instrument prefect worker start -p <your-pool>

- If you run flows as scripts locally, wrap those too:

opentelemetry-instrument python path/to/your_flow.py

- Docker Compose ``` services: prefect-worker: image: prefecthq/prefect:3-latest command: > sh -c "pip install opentelemetry-distro opentelemetry-exporter-otlp &&

opentelemetry-bootstrap -a install && opentelemetry-instrument prefect worker start -p my-pool" environment: - PREFECT_API_URL=http://prefect-server:4200/api - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 - OTEL_TRACES_EXPORTER=otlp - OTEL_METRICS_EXPORTER=otlp - OTEL_SERVICE_NAME=prefect-worker depends_on: - prefect-server - otel-collector

- Kubernetes
  - In the worker Deployment (or Job), wrap the command and add OTEL_* env vars, similar to the server example:

containers: - name: prefect-worker image: prefecthq/prefect:3-latest env: - name: PREFECT_API_URL value: http://prefect-server:4200/api - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://otel-collector:4317 - name: OTEL_TRACES_EXPORTER value: otlp - name: OTEL_METRICS_EXPORTER value: otlp - name: OTEL_SERVICE_NAME value: prefect-worker command: ["sh", "-c"] args: - > pip install opentelemetry-distro opentelemetry-exporter-otlp && opentelemetry-bootstrap -a install && opentelemetry-instrument prefect worker start -p my-pool ``` Optional: expose simple client metrics from Prefect processes - Prefect 3 has a small “client metrics” server that can be enabled for Prometheus-style scraping. If you want that: - Enable and set port via settings (env names may vary by version; see settings ref): - enabled: likely via

PREFECT_CLIENT_METRICS_ENABLED=true

- port: likely via

PREFECT_CLIENT_METRICS_PORT=4201

- Then point your Prometheus (or an OTel Collector with prometheus receiver) to scrape the process on that port. - See settings reference here: Prefect 3 settings reference - Note: This metrics server is separate from OTel exporters; it’s just another option if scraping is simpler in your environment. Notes and expectations - Auto-instrumentation will cover FastAPI/Uvicorn for the Server and common libraries (HTTP clients, DB drivers, etc.) used by your flows. You can enrich spans in flows/tasks with custom attributes if you like. - Logs via OTel are supported but vary by backend; traces are the most mature signal. - There isn’t a Prefect-specific OTEL toggle for the server today; the recommended path is standard OpenTelemetry instrumentation as shown above. References - Prefect Server concepts: Server concepts - Telemetry APIs: - prefect.telemetry.run_telemetry - prefect.telemetry.services - prefect.telemetry.bootstrap - Settings reference (for env vars): Prefect 3 settings reference Want me to translate this into concrete manifests or compose snippets for your exact environment and backend (e.g., Datadog agent, Tempo, or a specific OTel Collector config)?

Thanks for sharing the error. That one isn’t Prefect-specific — it’s a Python packaging issue. Python 3.12 removed stdlib distutils, and the “distutils-precedence.pth” file (installed by setuptools) tries to import

_distutils_hack

. If

setuptools

isn’t installed (or is broken), you’ll see:

ModuleNotFoundError: No module named '_distutils_hack'

Quick fix inside your Prefect container - Ensure setuptools is installed in the same environment where you’re running Prefect (your path shows /opt/conda/envs/prefect): Option A: pip

/opt/conda/envs/prefect/bin/python -m pip install --upgrade pip setuptools wheel

Option B: conda (if conda/mamba is available in the image)

conda install -n prefect -c conda-forge setuptools
# or
mamba install -n prefect -c conda-forge setuptools

Notes - This error often appears at Python startup but can still let the process continue; still, you should fix it or you’ll hit other packaging issues later (e.g., installing extras, OTel libs). - Do not remove the

distutils-precedence.pth

file as a “fix”; reinstalling

setuptools

is the correct solution. Deleting the file only hides the symptom and can cause other issues. - If you’re installing additional packages at runtime (e.g., for OpenTelemetry), prefer doing it in a custom image layer so the environment remains consistent. If you want, I can share a small Dockerfile snippet to bake the fix (and any other dependencies) into your Prefect image. Which Prefect image/tag are you using and how are you starting the container?