Dolor Oculus
11/18/2020, 10:18 PMlog_stdout=True
to all of our tasks, and the log messages are showing up just fine in the agent's stdout, but we'd like them to also be available on the web ui. We've discovered that by getting the logger from the context, we can do this, but this is a pretty invasive change for our codebase (most of our modules are using standard logging and we don't want to change that). Is there any magic we can invoke to get our logger messages to show up on the web ui?Jim Crist-Harif
11/18/2020, 10:21 PMlog_stdout=True
print statements from tasks show up as logs locally, but aren't visible in the webui?Dolor Oculus
11/18/2020, 10:23 PMlogging
import logging
_LOGGER = logging.getLogger(__name__)
<http://_LOGGER.info|_LOGGER.info>("I'm a log message")
Jim Crist-Harif
11/18/2020, 10:24 PMlog_stdout
to properly pick up logs generated via other loggers within your tasks? This won't be the case, the way python's logging `StreamHandler`s work our stdout redirection won't properly catch these. You're only seeing things work locally because if you stream local logs through the local agent you're actually just streaming all process output, which is different.Dolor Oculus
11/18/2020, 10:24 PMJim Crist-Harif
11/18/2020, 10:26 PMsys.stdout
early, meaning that redirecting it in a task won't actually work.sys.stdout
, if a library grabs a reference to sys.stdout
before we can do that, the redirection won't work.Dolor Oculus
11/18/2020, 10:29 PMJim Crist-Harif
11/18/2020, 10:48 PMextra_loggers
configuration. Loggers with names specified in this list will be configured to log to prefect. Since loggers are hierarchical (and assuming the libraries are using __name__
for the logger names) this would be a list of package names:
# in your ~/.prefect/config.toml
[logging]
extra_loggers = ['package1', 'package2']
Dolor Oculus
11/18/2020, 10:55 PMloggers = [logging.getLogger(name) for name in logging.root.manager.loggerDict]
though ๐Jim Crist-Harif
11/18/2020, 10:57 PMPREFECT__LOGGING__EXTRA_LOGGERS="['package1', 'package2', ...]"
if that's easier. Doing it programmatically is fine, but won't transfer to other processes if you're e.g. using a DaskExecutor
.Dolor Oculus
11/18/2020, 10:58 PMJim Crist-Harif
11/18/2020, 11:01 PMDolor Oculus
11/18/2020, 11:01 PMJim Crist-Harif
11/18/2020, 11:04 PMloggerDict
, just the root parents, e.g. foo
but not foo.bar
. Most python libraries I interact with don't use logging, I'm surprised the list you do is so large you can't statically configure it once.Dolor Oculus
11/18/2020, 11:05 PMJim Crist-Harif
11/18/2020, 11:16 PMChris White
Marvin
11/18/2020, 11:31 PM