Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Apologies is this is just my lack of knowledge on pytest but is there a way to use Prefect’s existing logging (with customization - e.g. flow_name, task_name, etc) within the context of a pytest test?

when I ran pytest with a test that defined and ran a flow, the logging output wasn’t the same as when I run a normally. I have log_cli=true for pytest config, but it seems like maybe pytest is overriding the prefect logger behavior

Hm interesting. You could take a look at our conftest.py - pytest will capture logs and display them in its own format but it will also capture stderr which will have logs in our format

We definitely have flow run logs in our format during internal tests.

There is definitely some subideal logging behaviour overwrite happening when running Prefect :confused: see also this thread: <https://discourse.prefect.io/t/stdout-not-logged-with-log-prints-true/2818>

I mean… you want us to override `sys.stdout` with a duplexing handler?

`log_prints` is pretty explicitly talking about “prints” because it’s safer for us to patch

<@U01CEUST9B5> in the context of this thread I was more referring to the fact of my post that when I added my own logging handler in a package it was somehow overwritten and not called in the context of a flow or task - in the a normal python execution context it was called - and imho that shouldn’t happen :slightly_smiling_face:

regarding the actual problem with `sys.stdout` I believe we would profit a lot if it was at least an option, but have to dig deeper into that - as many tools provide some form of `stdout` binding (e.g. subprocesses) it would be vital to us so we can use python packages as they come - I am thinking something like this as an option: <https://stackoverflow.com/questions/19425736/how-to-redirect-stdout-and-stderr-to-logger-in-python/31688396#31688396>

Unfortunately Python logging doesn't really support incremental updates well and we need to take control of logging to provide the UX our users want. We provide full control over the logging configuration so you can usually do what you need. If you want to dynamically create handlers support is complicated, if you open a GitHub issue we can try.

We could support stdout but it'd probably need to be a global setting rather than a task / flow option because it's harder to constrain to a specific scope in the way we do for log prints.

Will look deeper into this, thanks <@U01CEUST9B5> for your responses!