https://prefect.io logo
Title
d

Dave

11/18/2022, 3:19 PM
Hello, thank you for the amazing work on Prefect! I'm wondering whether there's a simple way to set the logLevel for all tasks and flows within Python. Specifically, I'm seeking a way to do this that doesn't leverage logging configuration files or use
get_run_logger().setLevel(level)
within each task and flow. Would this require overriding flow and task initiation, or similar? Thank you in advance for any help!
1
r

redsquare

11/18/2022, 3:21 PM
you can use PREFECT_LOGGING_LEVEL env var
👍 1
d

Dave

11/18/2022, 3:24 PM
Thanks @redsquare! When I've attempted to use the environment variable it sometimes appears to be ignored or disregarded. My usecase involves importing a Prefect flow as a package for use within sometimes unpredictable environments.
r

Ryan Peden

11/18/2022, 3:26 PM
You can also customize by creating a custom logging configuration based on this example on GitHub. This section of the docs discusses a couple of ways to get Prefect to use your custom logging config.
You'll see in that file that you can set different levels per logger, so you might find that an ideal way to get exactly the log levels you need
That logging.yml file is also the default Prefect uses. It looks like the various loggers get their levels from a couple of different environment variables, which might explain the behavior you're seeing when you set
PREFECT_LOGGING_LEVEL
. Some loggers use that, and some use
PREFECT_LOGGING_SERVER_LEVEL
.
👍 1
d

Dave

11/18/2022, 3:38 PM
Thank you @Ryan Peden! My latest attempts used the following block to attempt to force environment variable-based logging acknowledgement. I noticed that direct python execution portrayed no logs correctly, but that other environments (conda / jupyter lab) refused to obey the logging settings. I missed the
PREFECT_LOGGING_SERVER_LEVEL
, which might explain the inconsistent behavior, unsure. I'll give that a try. Open to other suggestions as well.
import os
os.environ["PREFECT_LOGGING_LEVEL"] = log_level
os.environ["PREFECT_LOGGING_ROOT_LEVEL"] = log_level
os.environ["PREFECT_LOGGING_HANDLERS_CONSOLE_LEVEL"] = log_level
os.environ["PREFECT_LOGGING_HANDLERS_CONSOLE_FLOW_RUNS_LEVEL"] = log_level
os.environ["PREFECT_LOGGING_HANDLERS_CONSOLE_TASK_RUNS_LEVEL"] = log_level
r

Ryan Peden

11/18/2022, 3:53 PM
Setting them via
os.environ
may or may not work depending on when you run those commands. I think it will work if you run them before you import Prefect. Otherwise, I believe Prefect reads the settings when its module initializes - but I'm going to go double check to code to verify that.
🙌 1
d

Dave

11/18/2022, 4:10 PM
Thanks! In testing again using the
PREFECT_LOGGING_SERVER_LEVEL
environment variable forced within Python things seemed to work as expected. Appreciate the help!
r

Ryan Peden

11/18/2022, 4:10 PM
You're welcome! I'm happy to hear that does what you need.