Hey guys, I would like to instrument my self hosted Prefect deployment with

OpenTelemetry

(Its the last part of our micro-service architecture thats missing it) I was looking online and only managed to find a single old thread and one attempt that doesn't look working/maintained. Anyone tried this? Know if its doable even? • It would require passing the Open Telemetry trace id from a parent flow to each of its subflows/tasks. Prefect run context which might be an option for this does not appear to allow additional fields. • The tasks/flows themselves would also need to be instrumented of course Would appreciate any help 🙏 Thanks all

Interesting! We use OpenTelemetry for internal parts of Cloud, and we’ve kicked around the idea of deeper integration with our client and OSS code. If I were you, I would create a wrapper entrypoint for your uses of Prefect, set up automatic OTEL instrumentation, and then use that wrapper for CLI commands (like starting the API server). Here’s an example wrapper I’ve played with — I didn’t hook this up to anything to verify the instrumentation:

import sys

from opentelemetry.instrumentation.asyncpg import AsyncPGInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

from prefect.cli import app


def run():
    AsyncPGInstrumentor().instrument()
    HTTPXClientInstrumentor().instrument()

    # Just FYI, the default settings for this one add comments to all SQL
    # statements to include a trace ID, which is a bit noisy.
    SQLAlchemyInstrumentor().instrument()

    # Set up OTEL span collectors, exporters, etc.

    # Pass through commands to the Prefect CLI.
    app(sys.argv[1:])


if __name__ == "__main__":
    run()

The one that’s hard is FastAPI instrumentation, which requires adding the instrumentation as FastAPI middleware. You’d have to have access to the FastAPI application object that we create when we start the server. There isn’t a hook that would give you access to this object currently, nor does it appear to be possible to patch our

create_app

function with something like

mock

due to the way uvicorn imports the function. (Or there is and I just can’t figure it out. 😄) If you maintain a fork of Prefect, you can modify the code directly to add the OTEL instrumentation middleware. You’d add it here, in a second

add_middleware

call: https://github.com/PrefectHQ/prefect/blob/d54ae148cfe98ecd4d535ec2186821bb54730ee3/src/prefect/server/api/server.py You can see an example of how to do this in the OTEL docs.

@Andrew Brookins I really appreciate the detailed responses! Thanks I'll start with the main auto-instrumentation as you suggested, and then move on to flows and tasks (possibly even manually passing in the TraceID as just another argument for now). I'll also be sure to update here if I get anything working that can be of use to others. (Also, purely based on OTEL adoption, I think this might be worthwhile for you as a company to look into, as you mentioned) Thanks again!

Welcome! I think we’re in very different timezones, but LMK if you want to hang out and pair on this sometime.

Wow thanks. I'll take a serious crack at it first so I don't waste your time, but again, I super appreciate the support and might just take you up on it if I get stuck 🙂

auth