Hey slightly smiling face I have shared code that is used by Prefect Community #ask-community

Hey :slightly_smiling_face: I have shared code tha...

Nico Neumann

12/11/2022, 9:45 PM

Hey 🙂 I have shared code that is used by both prefect flows + system where no prefect is installed. Thus I need a logger which is able to log everywhere. I created an own logger but have some problems with prefect to get it running. I only found documentation on how to set up a custom logger for v1: https://docs-v1.prefect.io/core/concepts/logging.html#extra-loggers But I think it should work the same for v2. I also had a look here: https://discourse.prefect.io/t/prefect-logging-faq/1476

✅ 1

Nico Neumann

12/11/2022, 9:45 PM

my_logger.py

Copy code

import logging
import sys

logger = logging.getLogger("my-logger")
logger.setLevel("DEBUG")
log_stream = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter("[%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s")
log_stream.setFormatter(formatter)
logger.addHandler(log_stream)

my_flow.py

Copy code

from prefect import flow
from my_logger import logger

def regular_log():
    print("Log inside a regular function")
    logger.debug("debug")
    <http://logger.info|logger.info>("info")
    logger.error("error")
    logger.warning("warning")
    logger.critical("critical")

@flow(name="my-flow")
def flow_log():
    print("Log inside a prefect flow")
    logger.debug("debug")
    <http://logger.info|logger.info>("info")
    logger.error("error")
    logger.warning("warning")
    logger.critical("critical")

if __name__ == "__main__":
    regular_log()
    print("-" * 30)
    flow_log()

Nico Neumann

12/11/2022, 9:45 PM

Using prefect >= 2.6.9 and calling

regular_log()

in the main function, I get an exception when trying to use my own logger. This also happens when I just define the import

from prefect import flow

and have no flow defined.

Copy code

raise MissingContextError(
prefect.exceptions.MissingContextError: Logger 'my-logger' attempted to send logs to Orion without a flow run id. The Orion log handler can only send logs within flow run contexts unless the flow run id is manually provided.

Using prefect 2.6.8 and calling

regular_log()

it does not crash. But in any cases I see duplicate logs calling

flow_log()

. Next to my own logger also prefect prints to logging:

Copy code

❯ PREFECT_LOGGING_EXTRA_LOGGERS=my-logger python my_flow.py
Log as regular function
[DEBUG 2022-12-11 22:40:53,949 my_flow.py:7] debug
[INFO 2022-12-11 22:40:53,949 my_flow.py:8] info
[ERROR 2022-12-11 22:40:53,949 my_flow.py:9] error
[WARNING 2022-12-11 22:40:53,949 my_flow.py:10] warning
[CRITICAL 2022-12-11 22:40:53,949 my_flow.py:11] critical
------------------------------
Log inside a prefect flow
22:40:54.366 | INFO    | prefect.engine - Created flow run 'spry-mayfly' for flow 'my-flow'
[DEBUG 2022-12-11 22:40:54,445 my_flow.py:16] debug
22:40:54.445 | DEBUG   | my-logger - debug
[INFO 2022-12-11 22:40:54,446 my_flow.py:17] info
22:40:54.446 | INFO    | my-logger - info
[ERROR 2022-12-11 22:40:54,446 my_flow.py:18] error
22:40:54.446 | ERROR   | my-logger - error
[WARNING 2022-12-11 22:40:54,446 my_flow.py:19] warning
22:40:54.446 | WARNING | my-logger - warning
[CRITICAL 2022-12-11 22:40:54,446 my_flow.py:20] critical
22:40:54.446 | CRITICAL | my-logger - critical
22:40:54.462 | INFO    | Flow run 'spry-mayfly' - Finished in state Completed()

Nico Neumann

12/11/2022, 9:46 PM

So in prefect >= 2.6.9 I cannot use the extra logger and in any version I see duplicate logs when using my logger inside a prefect flow. Do I need to set some configuration to avoid this?

Anna Geller

12/11/2022, 9:53 PM

you should be able to set the names of all custom loggers as comma-separated values on this setting:

Copy code

prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="logger1,logger2,logger3"

if this doesn't work as expected (e.g. you see duplicated logs), could you open a GitHub issue with a minimal reproducible example? alternatively, from 2.7 on, you can add

log_prints=True

on any decorator to log print statements

Nico Neumann

12/11/2022, 10:08 PM

Thanks @Anna Geller! I also tried

prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="my-logger"

, but it crashes with the same message on 2.6.9 - 2.7.1. And inside the flow I see the same duplicate messages. I tried

log_prints=True

and it works great 🙂 But I want to be able to distinguish between the different logging levels and also save the logs to a file. I will open a Github issue with the minimal example above.

🙌 1

Nico Neumann

12/11/2022, 10:27 PM

https://github.com/PrefectHQ/prefect/issues/7851

gratitude thank you 2

5 Views

Open in Slack

Previous Next