https://prefect.io logo
Title
m

Mars

08/17/2022, 2:03 PM
Hi, is there a way to export the flow logs from Prefect? I would like to save the logs for at least 90 days, preferably with a copy of the flow logs imported our own logging infrastructure (Loki) for searchability.
j

Jason Thomas

08/17/2022, 2:11 PM
We looked at doing this. Instead of exporting, we decided to add our own log handlers to write to our logs directly.
:gratitude-thank-you: 1
k

Khuyen Tran

08/17/2022, 2:54 PM
I’ll check with the team on this
m

Mars

08/17/2022, 3:01 PM
I think we can write our own logging function if necessary. The flows provide a bunch of context information like flow run ID, and we should be able to put this into a structured log format and output it to console. Our k8s pod loggers should pick it up from there.
👍 1
I still want logs to go to Prefect, too, so operators using the Prefect UI can view the logs right there, too. Our custom log function would be like
tee
, piping the logs to both locations (UI for operators, Loki for retention).
k

Khuyen Tran

08/17/2022, 3:09 PM
Just to make sure I understand you correctly, do you want to see the Prefect logs of your flow runs on Prefect UI?
a

Anna Geller

08/17/2022, 3:21 PM
@Mars this problem is on our radar - let me bring this to the product team Tackling this on a logger might be a solution but there could be a better approach
z

Zanie

08/17/2022, 3:44 PM
You should be able to customize the logging.yaml file!
:thank-you: 1
j

Jason Thomas

08/17/2022, 10:29 PM
@Mars, we’re doing what @Zanie suggests. Here’s our solution: • create log handler(s) • place a copy of
logging.yml
in
PREFECT_HOME
• customize the copy of
logging.yml
: â—¦ register our handlers in the
handlers
section (screenshot 1) â—¦ add handlers to the appropriate
loggers
(screenshot 2) • set
PREFECT_LOGGING_SETTINGS_PATH = "$PREFECT_HOME/logging.yml"
This does not change any of Prefect’s logging behavior - console and db output are the same. It just adds our handlers on top. I ended up doing a lot of customization to our handlers, but that’s because of some unusual requirements on our end. The first version was just writing to a local text file, and it was super easy. I didn’t use any third-party libraries for our handler, just made a subclass of
logging.Handler
. The python docs are pretty easy to follow. I don’t know what Loki is, but depending on the needs you can create your connection in the handler’s
__init__()
or in it’s
emit()
. Your handler can have it’s own formatter, or you can use one of the formatters in
logging.yml
- just add a
formatter
field to your handler’s registry entry. If Loki is text-file-based then you’re pretty much done. It is working very well for us.
:gratitude-thank-you: 1
z

Zanie

08/17/2022, 10:49 PM
Glad it’s working great for you 🙂
I could investigate a “merge” style option instead as well so you don’t need to copy our config.
j

Jason Thomas

08/17/2022, 10:52 PM
@Zanie I’ve been meaning to try that - see if I can just add the custom stuff in our copy and hoping it would pick up both. But I haven’t gotten around to it. That would be nice, because each time I upgrade I have to dig it up and make sure nothing’s changed
Basically, I don’t want to be responsible for maintaining Prefect’s loggers in my file
z

Zanie

08/17/2022, 10:54 PM
Yeah I agree. We expose it in this way so you can easily opt out of our loggers as well, but a friendlier interface will be nice to include in the long run.
:thank-you: 1
👍 1