Jesse
09/03/2025, 2:53 PMMarvin
09/03/2025, 2:53 PMMarvin
09/03/2025, 2:58 PMpip install "opentelemetry-sdk>=1.20" "opentelemetry-exporter-otlp>=1.20"
2) Point flow runs at your collector via environment variables
- Set the standard OTEL env vars on the work pool job template so every flow run inherits them. Example prefect.yaml
snippet:
deployments:
- name: send-otel
work_pool:
name: k8s-pool # or docker/process/etc.
job_variables:
env:
OTEL_EXPORTER_OTLP_ENDPOINT: "<http://otel-collector.default.svc:4317>"
OTEL_EXPORTER_OTLP_PROTOCOL: "grpc" # or "http/protobuf"
OTEL_SERVICE_NAME: "prefect-flows"
OTEL_RESOURCE_ATTRIBUTES: "service.namespace=data-platform,telemetry.distro=prefect"
# Optional auth/header(s)
# OTEL_EXPORTER_OTLP_HEADERS: "authorization=Bearer YOUR_TOKEN"
Docs on customizing job variables: Customize job variables
- This ensures the variables propagate regardless of worker type (Kubernetes/Docker/Process). If you prefer, you can also set these on the worker process itself so worker-level traces/metrics are exported too.
3) (Recommended) Initialize the OpenTelemetry SDK early so Prefect’s spans use your exporter
- Prefect uses OpenTelemetry under the hood. Setting the global tracer/meter provider at process start guarantees Prefect’s internal spans and your custom spans go to your collector.
- Put this at the top of your flow module (before defining/running flows), or import from a small `otel_setup.py`:
# Requires: opentelemetry-sdk, opentelemetry-exporter-otlp
import os
from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
# Set resource attributes; add your own org/team identifiers
resource = Resource.create({
"service.name": os.getenv("OTEL_SERVICE_NAME", "prefect-flows"),
"service.namespace": "data-platform",
})
# Tracing
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "<http://otel-collector:4317>"),
insecure=True # set accordingly for your endpoint
)
)
)
trace.set_tracer_provider(tracer_provider)
# Metrics (optional)
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "<http://otel-collector:4317>"),
insecure=True
)
)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
4) Add custom spans/metrics where helpful
- Custom spans inside a flow:
from prefect import flow, task
from opentelemetry import trace
@task
def work():
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("expensive-step") as span:
# annotate spans with useful attributes
span.set_attribute("component", "feature_x")
# do work...
return 1
@flow
def my_flow():
return work()
if __name__ == "__main__":
my_flow()
- Custom metrics:
```
from opentelemetry import metrics
meter = metrics.get_meter("prefect.custom")Marvin
09/03/2025, 2:58 PMOTEL_EXPORTER_OTLP_HEADERS
if your collector requires auth.
- Service naming: keep OTEL_SERVICE_NAME
stable so your backend aggregates correctly; add service.namespace
via OTEL_RESOURCE_ATTRIBUTES
.
- Workers vs flows: If you also want worker process telemetry, set the same env vars on the worker’s environment (e.g., the Kubernetes Deployment or Docker service that runs the worker).
Questions so I can tailor examples:
- Which work pool type are you using (Kubernetes, Docker, Process, ECS, etc.)?
- Which collector/backend and protocol (gRPC vs HTTP) do you prefer?
- Do you want telemetry from just flows/tasks, or also from the worker processes?Jesse
09/03/2025, 3:01 PMMarvin
09/03/2025, 3:01 PMMarvin
09/03/2025, 3:04 PMJesse
09/03/2025, 3:04 PMMarvin
09/03/2025, 3:04 PMMarvin
09/03/2025, 3:06 PMPREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY
controls Prefect Cloud’s own orchestration telemetry (internal to Prefect Cloud). It does not configure or interfere with OpenTelemetry/OTLP export to your collector.
- Default: true
- Effect: Enables/disables Prefect Cloud’s internal orchestration telemetry
- Not related to OTLP exporters; your OTEL_* env vars and SDK setup will still send traces/metrics to your collector regardless of this flag.
If you do want to turn it off:
PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=false
Docs: Settings reference (see CloudSettings: enable_orchestration_telemetry
)
If you share which work pool type you’re using, I can show exactly where to set this env var (and your OTEL_* vars) so it propagates to flow runs.