https://prefect.io logo
Title
n

Nico Neumann

12/11/2022, 9:45 PM
Hey 🙂 I have shared code that is used by both prefect flows + system where no prefect is installed. Thus I need a logger which is able to log everywhere. I created an own logger but have some problems with prefect to get it running. I only found documentation on how to set up a custom logger for v1: https://docs-v1.prefect.io/core/concepts/logging.html#extra-loggers But I think it should work the same for v2. I also had a look here: https://discourse.prefect.io/t/prefect-logging-faq/1476
1
my_logger.py
import logging
import sys

logger = logging.getLogger("my-logger")
logger.setLevel("DEBUG")
log_stream = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter("[%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s")
log_stream.setFormatter(formatter)
logger.addHandler(log_stream)
my_flow.py
from prefect import flow
from my_logger import logger

def regular_log():
    print("Log inside a regular function")
    logger.debug("debug")
    <http://logger.info|logger.info>("info")
    logger.error("error")
    logger.warning("warning")
    logger.critical("critical")

@flow(name="my-flow")
def flow_log():
    print("Log inside a prefect flow")
    logger.debug("debug")
    <http://logger.info|logger.info>("info")
    logger.error("error")
    logger.warning("warning")
    logger.critical("critical")

if __name__ == "__main__":
    regular_log()
    print("-" * 30)
    flow_log()
Using prefect >= 2.6.9 and calling
regular_log()
in the main function, I get an exception when trying to use my own logger. This also happens when I just define the import
from prefect import flow
and have no flow defined.
raise MissingContextError(
prefect.exceptions.MissingContextError: Logger 'my-logger' attempted to send logs to Orion without a flow run id. The Orion log handler can only send logs within flow run contexts unless the flow run id is manually provided.
Using prefect 2.6.8 and calling
regular_log()
it does not crash. But in any cases I see duplicate logs calling
flow_log()
. Next to my own logger also prefect prints to logging:
❯ PREFECT_LOGGING_EXTRA_LOGGERS=my-logger python my_flow.py
Log as regular function
[DEBUG 2022-12-11 22:40:53,949 my_flow.py:7] debug
[INFO 2022-12-11 22:40:53,949 my_flow.py:8] info
[ERROR 2022-12-11 22:40:53,949 my_flow.py:9] error
[WARNING 2022-12-11 22:40:53,949 my_flow.py:10] warning
[CRITICAL 2022-12-11 22:40:53,949 my_flow.py:11] critical
------------------------------
Log inside a prefect flow
22:40:54.366 | INFO    | prefect.engine - Created flow run 'spry-mayfly' for flow 'my-flow'
[DEBUG 2022-12-11 22:40:54,445 my_flow.py:16] debug
22:40:54.445 | DEBUG   | my-logger - debug
[INFO 2022-12-11 22:40:54,446 my_flow.py:17] info
22:40:54.446 | INFO    | my-logger - info
[ERROR 2022-12-11 22:40:54,446 my_flow.py:18] error
22:40:54.446 | ERROR   | my-logger - error
[WARNING 2022-12-11 22:40:54,446 my_flow.py:19] warning
22:40:54.446 | WARNING | my-logger - warning
[CRITICAL 2022-12-11 22:40:54,446 my_flow.py:20] critical
22:40:54.446 | CRITICAL | my-logger - critical
22:40:54.462 | INFO    | Flow run 'spry-mayfly' - Finished in state Completed()
So in prefect >= 2.6.9 I cannot use the extra logger and in any version I see duplicate logs when using my logger inside a prefect flow. Do I need to set some configuration to avoid this?
a

Anna Geller

12/11/2022, 9:53 PM
you should be able to set the names of all custom loggers as comma-separated values on this setting:
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="logger1,logger2,logger3"
if this doesn't work as expected (e.g. you see duplicated logs), could you open a GitHub issue with a minimal reproducible example? alternatively, from 2.7 on, you can add
log_prints=True
on any decorator to log print statements
n

Nico Neumann

12/11/2022, 10:08 PM
Thanks @Anna Geller! I also tried
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="my-logger"
, but it crashes with the same message on 2.6.9 - 2.7.1. And inside the flow I see the same duplicate messages. I tried
log_prints=True
and it works great 🙂 But I want to be able to distinguish between the different logging levels and also save the logs to a file. I will open a Github issue with the minimal example above.
🙌 1