Dolor Oculus11/18/2020, 10:18 PM
to all of our tasks, and the log messages are showing up just fine in the agent's stdout, but we'd like them to also be available on the web ui. We've discovered that by getting the logger from the context, we can do this, but this is a pretty invasive change for our codebase (most of our modules are using standard logging and we don't want to change that). Is there any magic we can invoke to get our logger messages to show up on the web ui?
Jim Crist-Harif11/18/2020, 10:21 PM
print statements from tasks show up as logs locally, but aren't visible in the webui?
Dolor Oculus11/18/2020, 10:23 PM
import logging _LOGGER = logging.getLogger(__name__) <http://_LOGGER.info|_LOGGER.info>("I'm a log message")
Jim Crist-Harif11/18/2020, 10:24 PM
to properly pick up logs generated via other loggers within your tasks? This won't be the case, the way python's logging `StreamHandler`s work our stdout redirection won't properly catch these. You're only seeing things work locally because if you stream local logs through the local agent you're actually just streaming all process output, which is different.
Dolor Oculus11/18/2020, 10:24 PM
Jim Crist-Harif11/18/2020, 10:26 PM
early, meaning that redirecting it in a task won't actually work.
, if a library grabs a reference to
before we can do that, the redirection won't work.
Dolor Oculus11/18/2020, 10:29 PM
Jim Crist-Harif11/18/2020, 10:48 PM
configuration. Loggers with names specified in this list will be configured to log to prefect. Since loggers are hierarchical (and assuming the libraries are using
for the logger names) this would be a list of package names:
# in your ~/.prefect/config.toml [logging] extra_loggers = ['package1', 'package2']
Dolor Oculus11/18/2020, 10:55 PM
loggers = [logging.getLogger(name) for name in logging.root.manager.loggerDict]
Jim Crist-Harif11/18/2020, 10:57 PM
if that's easier. Doing it programmatically is fine, but won't transfer to other processes if you're e.g. using a
PREFECT__LOGGING__EXTRA_LOGGERS="['package1', 'package2', ...]"
Dolor Oculus11/18/2020, 10:58 PM
Jim Crist-Harif11/18/2020, 11:01 PM
Dolor Oculus11/18/2020, 11:01 PM
Jim Crist-Harif11/18/2020, 11:04 PM
, just the root parents, e.g.
. Most python libraries I interact with don't use logging, I'm surprised the list you do is so large you can't statically configure it once.
Dolor Oculus11/18/2020, 11:05 PM
Jim Crist-Harif11/18/2020, 11:16 PM