Hi, is there a way to export the flow logs from Prefect? I would like to save the logs for at least 90 days, preferably with a copy of the flow logs imported our own logging infrastructure (Loki) for searchability.

We looked at doing this. Instead of exporting, we decided to add our own log handlers to write to our logs directly.

I’ll check with the team on this

I think we can write our own logging function if necessary. The flows provide a bunch of context information like flow run ID, and we should be able to put this into a structured log format and output it to console. Our k8s pod loggers should pick it up from there.

Just to make sure I understand you correctly, do you want to see the Prefect logs of your flow runs on Prefect UI?

@Mars this problem is on our radar - let me bring this to the product team Tackling this on a logger might be a solution but there could be a better approach

You should be able to customize the logging.yaml file!

@Mars, we’re doing what @Zanie suggests. Here’s our solution: • create log handler(s) • place a copy of

logging.yml

in

PREFECT_HOME

• customize the copy of

logging.yml

: ◦ register our handlers in the

handlers

section (screenshot 1) ◦ add handlers to the appropriate

loggers

(screenshot 2) • set

PREFECT_LOGGING_SETTINGS_PATH = "$PREFECT_HOME/logging.yml"

This does not change any of Prefect’s logging behavior - console and db output are the same. It just adds our handlers on top. I ended up doing a lot of customization to our handlers, but that’s because of some unusual requirements on our end. The first version was just writing to a local text file, and it was super easy. I didn’t use any third-party libraries for our handler, just made a subclass of

logging.Handler

. The python docs are pretty easy to follow. I don’t know what Loki is, but depending on the needs you can create your connection in the handler’s

__init__()

or in it’s

emit()

. Your handler can have it’s own formatter, or you can use one of the formatters in

logging.yml

- just add a

formatter

field to your handler’s registry entry. If Loki is text-file-based then you’re pretty much done. It is working very well for us.

Glad it’s working great for you 🙂

@Zanie I’ve been meaning to try that - see if I can just add the custom stuff in our copy and hoping it would pick up both. But I haven’t gotten around to it. That would be nice, because each time I upgrade I have to dig it up and make sure nothing’s changed

Yeah I agree. We expose it in this way so you can easily opt out of our loggers as well, but a friendlier interface will be nice to include in the long run.