Mars
08/17/2022, 2:03 PMJason Thomas
08/17/2022, 2:11 PMKhuyen Tran
08/17/2022, 2:54 PMMars
08/17/2022, 3:01 PMtee
, piping the logs to both locations (UI for operators, Loki for retention).Khuyen Tran
08/17/2022, 3:09 PMAnna Geller
08/17/2022, 3:21 PMZanie
08/17/2022, 3:44 PMJason Thomas
08/17/2022, 10:29 PMlogging.yml
in PREFECT_HOME
• customize the copy of logging.yml
:
â—¦ register our handlers in the handlers
section (screenshot 1)
â—¦ add handlers to the appropriate loggers
(screenshot 2)
• set PREFECT_LOGGING_SETTINGS_PATH = "$PREFECT_HOME/logging.yml"
This does not change any of Prefect’s logging behavior - console and db output are the same. It just adds our handlers on top.
I ended up doing a lot of customization to our handlers, but that’s because of some unusual requirements on our end. The first version was just writing to a local text file, and it was super easy.
I didn’t use any third-party libraries for our handler, just made a subclass of logging.Handler
. The python docs are pretty easy to follow. I don’t know what Loki is, but depending on the needs you can create your connection in the handler’s __init__()
or in it’s emit()
. Your handler can have it’s own formatter, or you can use one of the formatters in logging.yml
- just add a formatter
field to your handler’s registry entry. If Loki is text-file-based then you’re pretty much done.
It is working very well for us.Zanie
08/17/2022, 10:49 PMJason Thomas
08/17/2022, 10:52 PMZanie
08/17/2022, 10:54 PM