https://prefect.io logo
Title
y

Young Ho Shin

12/14/2022, 5:41 AM
Hello all. I have a question about logging: Is there a way to disable console/terminal logging while keeping logs visible on the prefect web UI? I thought something like
PREFECT_LOGGING_HANDLERS_CONSOLE_LEVEL=999 run_flow.py
would do the job, but this seems to turn off the logs on the web UI as well as the console.
👀 1
1
b

Bianca Hoch

12/14/2022, 8:50 PM
Hello Young, I was able to accomplish this by creating a modified logging.yml file, and storing it in
~/.prefect/
These are the contents of the file:
# Prefect logging config file.
#
# Any item in this file can be overridden with an environment variable:
#    `PREFECT_LOGGING_[PATH]_[TO]_[KEY]=VALUE`
#
# Templated values can be used to insert values from the Prefect settings at runtime.

version: 1
disable_existing_loggers: False

formatters:
    simple:
        format: "%(asctime)s.%(msecs)03d | %(message)s"
        datefmt: "%H:%M:%S"

    standard:
        (): prefect.logging.formatters.PrefectFormatter
        format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(name)s - %(message)s"
        flow_run_fmt: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Flow run %(flow_run_name)r - %(message)s"
        task_run_fmt: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Task run %(task_run_name)r - %(message)s"
        datefmt: "%H:%M:%S"

    json:
        class: prefect.logging.formatters.JsonFormatter
        format: "default"

# filters:
    # Define any custom filters to drops records containing
    # sensitive information
    # my_filter:
        # class: your_module.FilterClass

handlers:

    # The handlers we define here will output all logs they receieve by default
    # but we include the `level` so it can be overridden by environment

    orion:
        level: 0
        class: prefect.logging.handlers.OrionHandler

loggers:
    prefect:
        level: "${PREFECT_LOGGING_LEVEL}"

    prefect.extra:
        level: "${PREFECT_LOGGING_LEVEL}"
        handlers: [orion]

    prefect.flow_runs:
        level: NOTSET
        handlers: [orion]

    prefect.task_runs:
        level: NOTSET
        handlers: [orion]

    prefect.orion:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"

    prefect.client:
        level: "${PREFECT_LOGGING_LEVEL}"

    prefect.infrastructure:
        level: "${PREFECT_LOGGING_LEVEL}"

    uvicorn:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"

    fastapi:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"
Essentially, I removed the console handler completely from the original default yml file. This yml file is specific to prefect versions 2.6.9 and up.
This resulted in logs being disabled in the terminal, while still sending logs to cloud
Here's a test script if you'd like to try it out as well:
import logging
from prefect import flow, task, get_run_logger

@task
def test_task():
    logger = get_run_logger()
    <http://logger.info|logger.info>("abc")

@flow
def test_logging():
    test_task()

if __name__ == "__main__":
    test_logging()

#Deployment commands to enter in the terminal
#prefect deployment build ./disable_console_log.py:test_logging -n test-logging -q logging-queue
#prefect deployment apply ./test_logging-deployment.yaml
I am interested as to why you'd like to do this, though. 👀
y

Young Ho Shin

12/15/2022, 3:05 AM
@Bianca Hoch Hi Bianca. Thanks so much for your help. Your
logging.yml
does seem to work. I was also able to get the behavior I want (i.e. INFO levels logs on orion, but ERROR logs on console) by changing to
level: ERROR
for the console handler in the default yml file. I am still a bit confused why my original try of
PREFECT_LOGGING_HANDLERS_CONSOLE_LEVEL=999 run_flow.py
does not work. From the [docs](https://docs.prefect.io/concepts/logs/#logging-configuration) it seems like it should. I've tried
PREFECT_LOGGING_HANDLERS_CONSOLE_LEVEL=ERROR
and
PREFECT_LOGGING_HANDLERS_CONSOLE_LEVEL="ERROR"
, but this doesn't work either.
To answer your question about my use case: This may be more of a concern for me because my flow has ~5000 sub-tasks and the info-level logs were too verbose at the console, but still useful for debugging purposes so that I wanted them saved on orion. For example, I have a flow submission/monitoring script that displays flow/sub-task progress using tqdm while the flow is running and the logs were messing up the progress bar display.