Hey :slightly_smiling_face: I have shared code tha...
# prefect-community
Hey 🙂 I have shared code that is used by both prefect flows + system where no prefect is installed. Thus I need a logger which is able to log everywhere. I created an own logger but have some problems with prefect to get it running. I only found documentation on how to set up a custom logger for v1: https://docs-v1.prefect.io/core/concepts/logging.html#extra-loggers But I think it should work the same for v2. I also had a look here: https://discourse.prefect.io/t/prefect-logging-faq/1476
Copy code
import logging
import sys

logger = logging.getLogger("my-logger")
log_stream = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter("[%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s")
Copy code
from prefect import flow
from my_logger import logger

def regular_log():
    print("Log inside a regular function")

def flow_log():
    print("Log inside a prefect flow")

if __name__ == "__main__":
    print("-" * 30)
Using prefect >= 2.6.9 and calling
in the main function, I get an exception when trying to use my own logger. This also happens when I just define the import
from prefect import flow
and have no flow defined.
Copy code
raise MissingContextError(
prefect.exceptions.MissingContextError: Logger 'my-logger' attempted to send logs to Orion without a flow run id. The Orion log handler can only send logs within flow run contexts unless the flow run id is manually provided.
Using prefect 2.6.8 and calling
it does not crash. But in any cases I see duplicate logs calling
. Next to my own logger also prefect prints to logging:
Copy code
❯ PREFECT_LOGGING_EXTRA_LOGGERS=my-logger python my_flow.py
Log as regular function
[DEBUG 2022-12-11 22:40:53,949 my_flow.py:7] debug
[INFO 2022-12-11 22:40:53,949 my_flow.py:8] info
[ERROR 2022-12-11 22:40:53,949 my_flow.py:9] error
[WARNING 2022-12-11 22:40:53,949 my_flow.py:10] warning
[CRITICAL 2022-12-11 22:40:53,949 my_flow.py:11] critical
Log inside a prefect flow
22:40:54.366 | INFO    | prefect.engine - Created flow run 'spry-mayfly' for flow 'my-flow'
[DEBUG 2022-12-11 22:40:54,445 my_flow.py:16] debug
22:40:54.445 | DEBUG   | my-logger - debug
[INFO 2022-12-11 22:40:54,446 my_flow.py:17] info
22:40:54.446 | INFO    | my-logger - info
[ERROR 2022-12-11 22:40:54,446 my_flow.py:18] error
22:40:54.446 | ERROR   | my-logger - error
[WARNING 2022-12-11 22:40:54,446 my_flow.py:19] warning
22:40:54.446 | WARNING | my-logger - warning
[CRITICAL 2022-12-11 22:40:54,446 my_flow.py:20] critical
22:40:54.446 | CRITICAL | my-logger - critical
22:40:54.462 | INFO    | Flow run 'spry-mayfly' - Finished in state Completed()
So in prefect >= 2.6.9 I cannot use the extra logger and in any version I see duplicate logs when using my logger inside a prefect flow. Do I need to set some configuration to avoid this?
you should be able to set the names of all custom loggers as comma-separated values on this setting:
Copy code
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="logger1,logger2,logger3"
if this doesn't work as expected (e.g. you see duplicated logs), could you open a GitHub issue with a minimal reproducible example? alternatively, from 2.7 on, you can add
on any decorator to log print statements
Thanks @Anna Geller! I also tried
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="my-logger"
, but it crashes with the same message on 2.6.9 - 2.7.1. And inside the flow I see the same duplicate messages. I tried
and it works great 🙂 But I want to be able to distinguish between the different logging levels and also save the logs to a file. I will open a Github issue with the minimal example above.
🙌 1