Hi team, I’m a bit lost in how to configure JSON logging in Prefect 2.0. I see this concept document and this api reference on the JSONFormatter, but I can’t seem to find how to marry the two together. What we’re trying to achieve is: 1. Prefect Cloud 2.0 still shows logs as is 2. Kubernetes jobs output logs in JSON so that they can be ingested by datadog (we want each log entry to have the flow_name, task_name, etc. to make it easily searchable back in time).

All logging configurations is in

~/.prefect/logging.yml

. You can change the setting of the logging by typing:

prefect config set PREFECT_LOGGING_[PATH]_[TO]_[KEY]=VALUE

To change the formatter to json, you can type:

prefect config set PREFECT_LOGGING_HANDLERS_CONSOLE_FORMATTER=json

prefect config set PREFECT_LOGGING_HANDLERS_CONSOLE_TASK_RUNS_FORMATTER=json

prefect config set PREFECT_LOGGING_HANDLERS_CONSOLE_FLOW_RUNS_FORMATTER=json

Or simply change the value in

~/.prefect/logging.yml

handlers:

    # The handlers we define here will output all logs they receive by default
    # but we include the `level` so it can be overridden by environment

    console:
        level: 0
        class: logging.StreamHandler
        formatter: json

    console_flow_runs:
        level: 0
        class: logging.StreamHandler
        formatter: json

    console_task_runs:
        level: 0
        class: logging.StreamHandler
        formatter: json

    orion:
        level: 0
        class: prefect.logging.handlers.OrionHandler

Thank you @Khuyen Tran

Yes, you can set the environment variables (

environment

field) to be the above. For example,

{
  "PREFECT_LOGGING_HANDLERS_CONSOLE_FORMATTER": "json",
  "PREFECT_LOGGING_HANDLERS_CONSOLE_TASK_RUNS_FORMATTER": "json",
  "PREFECT_LOGGING_HANDLERS_CONSOLE_FLOW_RUNS_FORMATTER": "json"
}

Thanks @Khuyen Tran I will add this to the roadmap.

Thanks @Khuyen Tran. We’ve enabled that but it appears its not really sending valid JSON. It seems its sending a Python dictionary:

{'name': 'prefect.flow_runs', 'msg': "Finished in state Completed('All states completed.')", 'args': [], 'levelname': 'INFO', 'levelno': 20, 'pathname': '/usr/local/lib/python3.9/site-packages/prefect/engine.py', 'filename': 'engine.py', 'module': 'engine', 'exc_info': None, 'exc_text': None, 'stack_info': None, 'lineno': 366, 'funcName': 'begin_flow_run', 'created': 1666970625.7505996, 'msecs': 750.5996227264404, 'relativeCreated': 5440.329074859619, 'thread': 140168457979712, 'threadName': 'MainThread', 'processName': 'MainProcess', 'process': 7, 'flow_run_name': 'fancy-alpaca', 'flow_run_id': '61694402-3773-4e93-a24d-55318dbe0de1', 'flow_name': 'my_new_flow', 'send_to_orion': False}

Note the single quotes and the use of types like

False

and

None

great PR! Thanks for submitting one