Nico Neumann
12/11/2022, 9:45 PMimport logging
import sys
logger = logging.getLogger("my-logger")
logger.setLevel("DEBUG")
log_stream = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter("[%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s")
log_stream.setFormatter(formatter)
logger.addHandler(log_stream)
my_flow.py
from prefect import flow
from my_logger import logger
def regular_log():
print("Log inside a regular function")
logger.debug("debug")
<http://logger.info|logger.info>("info")
logger.error("error")
logger.warning("warning")
logger.critical("critical")
@flow(name="my-flow")
def flow_log():
print("Log inside a prefect flow")
logger.debug("debug")
<http://logger.info|logger.info>("info")
logger.error("error")
logger.warning("warning")
logger.critical("critical")
if __name__ == "__main__":
regular_log()
print("-" * 30)
flow_log()
regular_log()
in the main function, I get an exception when trying to use my own logger.
This also happens when I just define the import from prefect import flow
and have no flow defined.
raise MissingContextError(
prefect.exceptions.MissingContextError: Logger 'my-logger' attempted to send logs to Orion without a flow run id. The Orion log handler can only send logs within flow run contexts unless the flow run id is manually provided.
Using prefect 2.6.8 and calling regular_log()
it does not crash.
But in any cases I see duplicate logs calling flow_log()
. Next to my own logger also prefect prints to logging:
❯ PREFECT_LOGGING_EXTRA_LOGGERS=my-logger python my_flow.py
Log as regular function
[DEBUG 2022-12-11 22:40:53,949 my_flow.py:7] debug
[INFO 2022-12-11 22:40:53,949 my_flow.py:8] info
[ERROR 2022-12-11 22:40:53,949 my_flow.py:9] error
[WARNING 2022-12-11 22:40:53,949 my_flow.py:10] warning
[CRITICAL 2022-12-11 22:40:53,949 my_flow.py:11] critical
------------------------------
Log inside a prefect flow
22:40:54.366 | INFO | prefect.engine - Created flow run 'spry-mayfly' for flow 'my-flow'
[DEBUG 2022-12-11 22:40:54,445 my_flow.py:16] debug
22:40:54.445 | DEBUG | my-logger - debug
[INFO 2022-12-11 22:40:54,446 my_flow.py:17] info
22:40:54.446 | INFO | my-logger - info
[ERROR 2022-12-11 22:40:54,446 my_flow.py:18] error
22:40:54.446 | ERROR | my-logger - error
[WARNING 2022-12-11 22:40:54,446 my_flow.py:19] warning
22:40:54.446 | WARNING | my-logger - warning
[CRITICAL 2022-12-11 22:40:54,446 my_flow.py:20] critical
22:40:54.446 | CRITICAL | my-logger - critical
22:40:54.462 | INFO | Flow run 'spry-mayfly' - Finished in state Completed()
Anna Geller
12/11/2022, 9:53 PMprefect config set PREFECT_LOGGING_EXTRA_LOGGERS="logger1,logger2,logger3"
if this doesn't work as expected (e.g. you see duplicated logs), could you open a GitHub issue with a minimal reproducible example?
alternatively, from 2.7 on, you can add log_prints=True
on any decorator to log print statementsNico Neumann
12/11/2022, 10:08 PMprefect config set PREFECT_LOGGING_EXTRA_LOGGERS="my-logger"
, but it crashes with the same message on 2.6.9 - 2.7.1. And inside the flow I see the same duplicate messages.
I tried log_prints=True
and it works great 🙂
But I want to be able to distinguish between the different logging levels and also save the logs to a file.
I will open a Github issue with the minimal example above.