<@ULVA73B9P> How can I send OpenTelemetry logs to ...
# ask-marvin
s
@Marvin How can I send OpenTelemetry logs to Grafana from Prefect runs and server?
m
thought for 509.9 seconds
Short answer - Prefect 3.x emits OpenTelemetry traces for flow/task runs, but it does not export OpenTelemetry logs by default. - To get logs into Grafana via OTLP you need to add OpenTelemetry logging instrumentation (either with the “auto-instrument” wrapper or by configuring a log exporter programmatically). - For traces and logs to reach Grafana (Grafana Cloud or Grafana Agent/Alloy/OTel Collector), you must install the OTel SDK/exporters and set OTEL_* env vars. Below are practical, copy/paste steps for flows/workers and for Prefect Server. What Prefect does out of the box (Prefect 3.x) - Traces: Prefect creates spans for flow/task runs and state transitions. They will only be exported if you configure an OTLP exporter in your environment. - Logs: Prefect uses Python’s logging; there is no built-in OTel log export. You must enable OTel logging explicitly if you want logs via OTLP. Recommended setup - If your end goal is “logs in Grafana”, the simplest path is to forward stdout/stderr to Grafana Loki using Grafana Agent/Alloy (no OTel logging needed). If you specifically want OTel logs, use the steps below. Prereqs (install once in the environment where your flows/workers/server run)
Copy code
pip install 'prefect[otel]' 'opentelemetry-instrumentation-logging' 'opentelemetry-exporter-otlp'
Common environment variables (point at Grafana Alloy/Agent or an OTel Collector) - Change the endpoint/headers for Grafana Cloud per your stack’s instructions.
Copy code
# Identify the emitting service
export OTEL_SERVICE_NAME=prefect

# Export traces and logs via OTLP
export OTEL_TRACES_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp

# Where to send OTLP (gRPC 4317 or HTTP 4318)
export OTEL_EXPORTER_OTLP_ENDPOINT=<http://alloy:4317>
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# If sending to Grafana Cloud, also set headers (replace with your token/endpoint)
# export OTEL_EXPORTER_OTLP_ENDPOINT=https://<your-otlp-endpoint>
# export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <your_api_token>"

# Helpful when debugging
export OTEL_LOG_LEVEL=debug
Flows and workers (send both traces and logs) Option A: Easiest — run processes with OpenTelemetry’s auto-instrumentation wrapper - Local ad-hoc flow run
Copy code
export OTEL_SERVICE_NAME=prefect-flows
opentelemetry-instrument python my_flow.py
- Start a worker with instrumentation (polls your work pool and instruments spawned runs)
Copy code
export OTEL_SERVICE_NAME=prefect-worker
opentelemetry-instrument prefect worker start -p <your-pool-name>
Notes: - The auto-instrumentation wrapper enables the logging instrumentation without code changes and configures the SDK/exporters from your OTEL_* env vars. - If you’re using Docker/Kubernetes workers, install the same OTel packages in the image and wrap the entrypoint or command with
opentelemetry-instrument
. Alternatively, instrument programmatically in your flow entrypoint. Option B: Programmatic logging instrumentation (if you can’t use the wrapper) - At the top of your flow’s entrypoint module, initialize the OTel logs SDK and enable the logging instrumentor. Keep it minimal and let env vars drive the exporter. If you want this, I can share a version that matches your OTel SDK version; it’s a bit version-sensitive in Python. Prefect Server (self-hosted) traces and logs - Prefect Server can emit OTel telemetry (e.g., FastAPI/SQLAlchemy and Python logs) when the SDK/exporters are installed and enabled. - Start it under the auto-instrumentation wrapper so both traces and logs are exported:
Copy code
export OTEL_SERVICE_NAME=prefect-server
opentelemetry-instrument prefect server start
Traces only (no logs) - If you only want traces from flows/tasks, Prefect already creates spans. Install
opentelemetry-exporter-otlp
and set the OTEL_* vars (no logging instrumentation needed). Ensure orchestration telemetry is enabled:
Copy code
prefect config set cloud.enable_orchestration_telemetry=true
Where the signals go in Grafana
- Traces: go to Grafana Tempo (Grafana Cloud or self-hosted), typically via OTLP gRPC/HTTP to Grafana Alloy/OTel Collector. - Logs: go to Grafana Loki. You can send logs via OTel logs pipeline (OTLP receiver -> Loki exporter in Alloy/OTel Collector) or skip OTel logs entirely and scrape stdout with Alloy. If you choose OTel logs, configure your Alloy/Collector with an OTLP receiver and a Loki exporter for the logs pipeline. Examples for Alloy/OTel Collector - Minimal pipelines (pseudo): - Receivers: OTLP (grpc/http) - Processors: batch - Exporters: - traces → otlp to Grafana Cloud Tempo, or to Alloy/Tempo - logs → loki to Grafana Cloud Loki, or to Alloy/Loki If you’d like, share: - Where your flows/workers run (process, Docker, Kubernetes) - Whether you’re sending to Grafana Cloud or self-hosted Grafana/Alloy - Whether you want to instrument via the wrapper or in code I can then give you a concrete, copy/paste configuration for your worker base job template or container entrypoint and a matching Alloy/OTel Collector config. References - Prefect trace/telemetry implementation (source): prefect/telemetry/run_telemetry.py - OpenTelemetry Python (auto instrumentation): OpenTelemetry Python automatic instrumentation - Grafana Alloy (OTLP receiver and Loki/Tempo exporters): Grafana Alloy docs - Grafana Cloud (send OTLP data): Send OTLP data to Grafana Cloud