https://prefect.io logo
m

Moe

08/18/2023, 7:06 PM
Is there any documentation on how prefect's
logging.yaml
work? E.g. what python
logging.handlers
are available to specify? What attributes is it a must for us to specify for the handlers, formatters, loggers etc.? I tried using a simple regular FileHandler seems Prefect doesn't allow it because all my logs on UI disappeared and none written to local.
Copy code
handlers: 
    file:
      level: 0
      class: logging.handlers.FileHandler
      filename: C:\...\prefect.log
      formatter: standard
1
j

Jake Kaplan

08/18/2023, 7:20 PM
Hi you can modify logging.yaml to write to a file as well. It's not special to prefect, it's python logging configuration, heres an example:
Copy code
# Prefect logging config file.
#
# Any item in this file can be overridden with an environment variable:
#    `PREFECT_LOGGING_[PATH]_[TO]_[KEY]=VALUE`
#
# Templated values can be used to insert values from the Prefect settings at runtime.

version: 1
disable_existing_loggers: False

formatters:
    simple:
        format: "%(asctime)s.%(msecs)03d | %(message)s"
        datefmt: "%H:%M:%S"

    standard:
        (): prefect.logging.formatters.PrefectFormatter
        format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(name)s - %(message)s"
        flow_run_fmt: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Flow run %(flow_run_name)r - %(message)s"
        task_run_fmt: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Task run %(task_run_name)r - %(message)s"
        datefmt: "%H:%M:%S"

    debug:
        format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(threadName)-12s | %(name)s - %(message)s"
        datefmt: "%H:%M:%S"

    json:
        class: prefect.logging.formatters.JsonFormatter
        format: "default"

# filters:
    # Define any custom filters to drops records containing
    # sensitive information
    # my_filter:
        # class: your_module.FilterClass

handlers:

    # The handlers we define here will output all logs they receieve by default
    # but we include the `level` so it can be overridden by environment

    file:
        level: INFO
        class: logging.FileHandler
        formatter: standard
        filename: 'prefect.log'

    console:
        level: 0
        class: prefect.logging.handlers.PrefectConsoleHandler
        formatter: standard
        styles:
            log.web_url: bright_blue
            log.local_url: bright_blue

            log.info_level: cyan
            log.warning_level: yellow3
            log.error_level: red3
            log.critical_level: bright_red

            log.completed_state: green
            log.cancelled_state: yellow3
            log.failed_state: red3
            log.crashed_state: bright_red

            log.flow_run_name: magenta
            log.flow_name: bold magenta

    api:
        level: 0
        class: prefect.logging.handlers.APILogHandler

    
    debug:
        level: 0
        class: logging.StreamHandler
        formatter: debug

loggers:
    prefect:
        level: "${PREFECT_LOGGING_LEVEL}"
        handlers: [file]

    prefect.extra:
        level: "${PREFECT_LOGGING_LEVEL}"
        handlers: [api]

    prefect.flow_runs:
        level: NOTSET
        handlers: [api]

    prefect.task_runs:
        level: NOTSET
        handlers: [api]

    prefect.server:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"

    prefect.client:
        level: "${PREFECT_LOGGING_LEVEL}"

    prefect.infrastructure:
        level: "${PREFECT_LOGGING_LEVEL}"

    prefect._internal:
        level: "${PREFECT_LOGGING_INTERNAL_LEVEL}"
        propagate: false
        handlers: [debug]

    uvicorn:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"

    fastapi:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"

# The root logger: any logger without propagation disabled sends to here as well
root:
    # By default, we display warning level logs from any library in the console
    # to match Python's default behavior while formatting logs nicely
    level: WARNING
    handlers: [console]
the two changes are:
Copy code
handlers:
    file:
        level: INFO
        class: logging.FileHandler
        formatter: standard
        filename: 'prefect.log'
and
Copy code
loggers:
    prefect:
        level: "${PREFECT_LOGGING_LEVEL}"
        handlers: [file] <-- add the handler
m

Moe

08/18/2023, 7:21 PM
Just realized the above is not working because it should be
logging.FileHandler
for the regular but others such as timed handler are specified as
logging.handlers.TimedRotatingFileHandler
🙌 1