Gregory Hunt

08/14/2023, 2:27 PM
Good Morning Prefectarians! is it possible to log Json Structured logs to the consoles and have the standard logs show up in the UI?
@Nate how do I prefect to log structured ie. json logs in the server and worker consoles
Bumping, how to I get prefect to use the json logging for the server and worker console?


08/14/2023, 8:55 PM
im not sure i understand what you mean by the json logging here, do you have an example of what you want?

Gregory Hunt

08/14/2023, 8:56 PM
I was hoping for a more structured version of the logging
I saw a json format in the config file
but can't get it to log in the json format
I want to change the console logs to json, i foung this doc but it pre-dates the current logging.yaml format
I want to keep the api as is, but since we stream the console logs to a common log platform I was hoping to et them in json


08/14/2023, 9:49 PM
as mentioned in the lower part of that discourse post, you can change the logging.yaml file like e.g.
Copy code

    # The handlers we define here will output all logs they receieve by default
    # but we include the `level` so it can be overridden by environment

        level: 0
        class: prefect.logging.handlers.PrefectConsoleHandler
        formatter: json # <------ I made this change HERE
            log.web_url: bright_blue
            log.local_url: bright_blue

            log.info_level: cyan
            log.warning_level: yellow3
            log.error_level: red3
            log.critical_level: bright_red

            log.completed_state: green
            log.cancelled_state: yellow3
            log.failed_state: red3
            log.crashed_state: bright_red

            log.flow_run_name: magenta
            log.flow_name: bold magenta
and then set the logging settings path like
Copy code
prefect config set PREFECT_LOGGING_SETTINGS_PATH=logging.yml
and so the consoles output becomes JSON formatted like
Copy code
In [5]: @flow(log_prints=True)
   ...: def foo():
   ...:     print("Hello!")

In [6]: foo()
{"name":"prefect.engine","msg":"Created flow run 'careful-seagull' for flow 'foo'","args":[],"levelname":"INFO","levelno":20,"pathname":"/Users/nate/src/open-source/prefect-stuff/prefect/src/prefect/","filename":"","module":"engine","exc_info":null,"exc_text":null,"stack_info":null,"lineno":346,"funcName":"create_then_begin_flow_run","created":1692049727.434399,"msecs":434.0,"relativeCreated":116433.02083015442,"thread":6121811968,"threadName":"GlobalEventLoopThread","processName":"MainProcess","process":28969,"severity":"INFO"}

Gregory Hunt

08/15/2023, 12:51 PM