https://prefect.io logo
Channels
Powered by
o

oliland

09/07/2019, 1:44 PM
I'm confused by logging. It seems like Prefect configures its own logging at import time (ugh) and that it's configured to only log warning messages from tasks. How do I change this to ensure all INFO messages are displayed when running locally?
j

Jeremiah

09/07/2019, 1:52 PM
Hi @oliland - if you’re only seeing Warning messages, that may be a setting on your machine. Prefect is configured to show 
INFO
messages by default, as you can see here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml#L26. If you need to change this directly, you can set the environment variable 
PREFECT__LOGGING__LEVEL=INFO
o

oliland

09/07/2019, 1:53 PM
I gave that a try but still no joy. Here's a minimal repro of what I'd like to do:
Copy code
logger = logging.getLogger(__name__)

@task(name="Test logging")
def test_logging():
    <http://logger.info|logger.info>("Hidden by default")
    logger.warning("Shown by default")

with Flow("test") as flow:
    test_logging()
flow.run()
I understand I could grab a reference to the prefect logger through 
prefect.context
- but I can't really inject the prefect logger all the way down the stack into the methods I'm calling.
j

Jeremiah

09/07/2019, 1:56 PM
Ah, you’re expecting Prefect to change your global logging settings. It only affects loggers within the prefect namespace. Use our logging utility to grab one of those loggers, like this:
Copy code
from prefect import task, Flow
import prefect.utilities

logger = prefect.utilities.logging.get_logger(__name__)

@task(name="Test logging")
def test_logging():
    <http://logger.info|logger.info>("Hidden by default")
    logger.warning("Shown by default")

with Flow("test") as flow:
        test_logging()
flow.run()
If you define your 
Task
as a class, rather than with the 
@task
decorator, you can also use 
self.logger
I’ll make sure we add a doc about how to access the loggers
o

oliland

09/07/2019, 2:03 PM
Thanks for the help - I'm still a bit lost though, as that was just a toy example. Here's a better example:
Copy code
import mypackage


@task(name="Test logging")
def test_logging():
    my_package.run()

with Flow("test") as flow:
    test_logging()
flow.run()
with this setup, the prefect logger spits out 
WARNING
messages from 
mypackage
- but not 
INFO
messages. How can I change that?
j

Jeremiah

09/07/2019, 2:06 PM
I’m afraid I can’t say without seeing all the code. As long as you’re using a logger generated by 
prefect.utilities.get_logger
, it will respect the log setting in 
prefect.config.logging.level
Otherwise, it will just use your global Python log settings, which I believe default to Warning and higher
o

oliland

09/07/2019, 2:08 PM
is there a way to disable all of prefect's log handlers? as calling 
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)
solves this issue for me, but it means I get double log entries
alternatively you mention the default is warning or higher - which config are you referring to?
j

Jeremiah

09/07/2019, 2:09 PM
Python’s default is Warning+, so if you’re using any Python logger without modifying 
logging.basicConfig
you’ll only get warning and higher
A quick way to disable Prefect’s handlers would be to access the logger returned by 
prefect.utilities.logging.get_logger()
and set its 
handlers
attribute to 
[]
. I don’t think we expose a first-class way to disable them right now
o

oliland

09/07/2019, 2:12 PM
right - I normally call 
basicConfig
myself - so I guess I'm just missing a way to change the default
j

Jeremiah

09/07/2019, 2:12 PM
Got it. I think what we actually should consider is a way to let Prefect control global logging if the user wants
so setting that one variable would take care of all of this for you
but only on an opt-in basis
o

oliland

09/07/2019, 2:19 PM
this works for me though - thanks for the help!
Copy code
prefect.utilities.logging.get_logger().handlers = []
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)
j

Jeremiah

09/07/2019, 2:22 PM
👌 Just PR’d a first cut at a Logging concept doc, will follow up with more this week!
😎 1
Powered by