Kevin Weiler
12/10/2020, 8:13 PMPREFECT__LOGGING__EXTRA_LOGGERS
environment variable both when launching my agent and injecting that variable into my docker container at runtime (with the -env
arg). Any advice on how to proceed?Zanie
Kevin Weiler
12/10/2020, 8:15 PMPREFECT__LOGGING__EXTRA_LOGGERS="['research']"
research
is the name of the logger--env PREFECT__LOGGING__EXTRA_LOGGERS=${PREFECT__LOGGING__EXTRA_LOGGERS}
Zanie
Kevin Weiler
12/10/2020, 8:18 PMresearch/flow.py
Zanie
log_stdout
flag to your task https://docs.prefect.io/core/concepts/logging.html#logging-stdout
• Consider just using <http://prefect.context.logger.info|prefect.context.logger.info>("…")
If those don’t work/fit your use-case I can take a look again tomorrow morning!Kevin Weiler
12/10/2020, 11:01 PMZanie
Kevin Weiler
12/10/2020, 11:07 PMlog_stdout=True
to the task decorator … no dice:
https://github.com/aquanauts/prefect_logger_repro/commit/4fd61a863c2bbeeb9f313278430c38f52a42a26a#Zanie
import os
import sys
import logging
import prefect
from prefect import task, Flow
def get_logger():
logger = logging.getLogger("my-named-logger")
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.setLevel(logging.DEBUG)
return logger
@task()
def log_message():
logger = get_logger()
<http://logger.info|logger.info>("Hello world!")
logger.error("Foo!")
with Flow('flow-with-extra-logger') as flow:
log_message()
flow.register("first")
Agent run with: prefect agent local start --env PREFECT__LOGGING__EXTRA_LOGGERS="['my-named-logger']"
flow.storage = Docker()
flow.register("first")
and using the docker agent works as wellKevin Weiler
12/11/2020, 7:12 PMresearch
because that’s the package nameZanie
my-named-logger
in my example instead of research
Kevin Weiler
12/11/2020, 7:15 PMZanie
import os
import sys
import logging
import prefect
from prefect import task, Flow
from prefect.environments.storage import Docker
LOGGER = logging.getLogger("my-named-logger")
LOGGER.addHandler(logging.StreamHandler(sys.stdout))
LOGGER.setLevel(logging.DEBUG)
@task()
def log_message():
<http://LOGGER.info|LOGGER.info>("Hello world!")
LOGGER.error("Foo!")
with Flow('flow-with-extra-logger') as flow:
log_message()
flow.storage = Docker()
flow.register("first")
Kevin Weiler
12/11/2020, 8:49 PM_LOGGER
is used in the task definition. But if I import a function that has _LOGGER
defined in it’s file - I don’t get the outputZanie
❯ python research/flow.py
Traceback (most recent call last):
File "research/flow.py", line 6, in <module>
from research.example_pkg import things
ModuleNotFoundError: No module named 'research'
PYTHONPATH
not PYTHON_PATH
haha — I should be able to run your codeKevin Weiler
12/11/2020, 9:43 PMPYTHONPATH
but it’s what happened (with this repo anyway)Zanie
LocalDaskExecutor(scheduler="processes")
when I switch to using a LocalExecutor
the logs work as expected!LocalDaskExecutor()
without using the processes scheduler worksKevin Weiler
12/11/2020, 9:56 PMZanie
Jim Crist-Harif
12/11/2020, 10:01 PMKevin Weiler
12/11/2020, 10:06 PMextra_loggers
doesn’t work as expected when using LocalDaskExecutor(scheduler="processes")
import sys
import logging
from prefect import task, Flow
from prefect.engine.executors import LocalDaskExecutor
from prefect.environments import LocalEnvironment
LOGGER = logging.getLogger("my-named-logger")
LOGGER.addHandler(logging.StreamHandler(sys.stdout))
LOGGER.setLevel(logging.DEBUG)
@task()
def log_message():
<http://LOGGER.info|LOGGER.info>("Hello world!")
LOGGER.error("Foo!")
with Flow('flow-with-extra-logger') as flow:
log_message()
flow.environment = LocalEnvironment(executor=LocalDaskExecutor(scheduler="processes"))
flow.run()
PREFECT__LOGGING__EXTRA_LOGGERS="['my-named-logger']"
- this example does not log the expected output to the GUIJim Crist-Harif
12/11/2020, 10:16 PMKevin Weiler
12/11/2020, 10:18 PMJim Crist-Harif
12/11/2020, 10:19 PMKevin Weiler
12/11/2020, 10:35 PMDylan
Kevin Weiler
01/11/2021, 8:31 PMDolor Oculus
01/22/2021, 3:58 AM