https://prefect.io logo
Title
t

Tomer Friedman

05/15/2023, 12:10 PM
Hey guys, I would like to instrument my self hosted Prefect deployment with
OpenTelemetry
(Its the last part of our micro-service architecture thats missing it) I was looking online and only managed to find a single old thread and one attempt that doesn't look working/maintained. Anyone tried this? Know if its doable even? • It would require passing the Open Telemetry trace id from a parent flow to each of its subflows/tasks. Prefect run context which might be an option for this does not appear to allow additional fields. • The tasks/flows themselves would also need to be instrumented of course Would appreciate any help 🙏 Thanks all
a

Andrew Brookins

05/15/2023, 11:39 PM
Interesting! We use OpenTelemetry for internal parts of Cloud, and we’ve kicked around the idea of deeper integration with our client and OSS code. If I were you, I would create a wrapper entrypoint for your uses of Prefect, set up automatic OTEL instrumentation, and then use that wrapper for CLI commands (like starting the API server). Here’s an example wrapper I’ve played with — I didn’t hook this up to anything to verify the instrumentation:
import sys

from opentelemetry.instrumentation.asyncpg import AsyncPGInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

from prefect.cli import app


def run():
    AsyncPGInstrumentor().instrument()
    HTTPXClientInstrumentor().instrument()

    # Just FYI, the default settings for this one add comments to all SQL
    # statements to include a trace ID, which is a bit noisy.
    SQLAlchemyInstrumentor().instrument()

    # Set up OTEL span collectors, exporters, etc.

    # Pass through commands to the Prefect CLI.
    app(sys.argv[1:])


if __name__ == "__main__":
    run()
The one that’s hard is FastAPI instrumentation, which requires adding the instrumentation as FastAPI middleware. You’d have to have access to the FastAPI application object that we create when we start the server. There isn’t a hook that would give you access to this object currently, nor does it appear to be possible to patch our
create_app
function with something like
mock
due to the way uvicorn imports the function. (Or there is and I just can’t figure it out. 😄) If you maintain a fork of Prefect, you can modify the code directly to add the OTEL instrumentation middleware. You’d add it here, in a second
add_middleware
call: https://github.com/PrefectHQ/prefect/blob/d54ae148cfe98ecd4d535ec2186821bb54730ee3/src/prefect/server/api/server.py You can see an example of how to do this in the OTEL docs.
I haven’t tried instrumenting flows or tasks, but I expect that after you had automatic instrumentation set up, you could experiment with a
Flow
or
Task
subclass that added instrumentation by wrapping calls in OTEL spans.
Oh, for completion’s sake, another option you have is creating your own
start
CLI wrapper, where you’d need an alternative
create_app
implementation (that you can point to in dot notation, i.e.,
your.module.create_app
, which itself attaches the middleware.
t

Tomer Friedman

05/16/2023, 10:41 AM
@Andrew Brookins I really appreciate the detailed responses! Thanks I'll start with the main auto-instrumentation as you suggested, and then move on to flows and tasks (possibly even manually passing in the TraceID as just another argument for now). I'll also be sure to update here if I get anything working that can be of use to others. (Also, purely based on OTEL adoption, I think this might be worthwhile for you as a company to look into, as you mentioned) Thanks again!
a

Andrew Brookins

05/16/2023, 4:25 PM
Welcome! I think we’re in very different timezones, but LMK if you want to hang out and pair on this sometime.
t

Tomer Friedman

05/17/2023, 7:43 AM
Wow thanks. I'll take a serious crack at it first so I don't waste your time, but again, I super appreciate the support and might just take you up on it if I get stuck 🙂
🙌 1
j

Josep Batallé Oronich

05/17/2023, 10:41 AM
auth