https://prefect.io logo
Title
c

Chris Goddard

04/25/2023, 10:22 PM
Apologies is this is just my lack of knowledge on pytest but is there a way to use Prefect’s existing logging (with customization - e.g. flow_name, task_name, etc) within the context of a pytest test?
z

Zanie

04/25/2023, 11:31 PM
I'm not quite sure what you mean
c

Chris Goddard

04/25/2023, 11:33 PM
when I ran pytest with a test that defined and ran a flow, the logging output wasn’t the same as when I run a normally. I have log_cli=true for pytest config, but it seems like maybe pytest is overriding the prefect logger behavior
z

Zanie

04/26/2023, 12:15 AM
Hm interesting. You could take a look at our conftest.py - pytest will capture logs and display them in its own format but it will also capture stderr which will have logs in our format
We definitely have flow run logs in our format during internal tests.
c

Chris Goddard

04/26/2023, 3:09 AM
thanks I’ll take a look
m

Maximilian Schulz

04/27/2023, 7:06 PM
There is definitely some subideal logging behaviour overwrite happening when running Prefect 😕 see also this thread: https://discourse.prefect.io/t/stdout-not-logged-with-log-prints-true/2818
z

Zanie

04/27/2023, 7:33 PM
I mean… you want us to override
sys.stdout
with a duplexing handler?
log_prints
is pretty explicitly talking about “prints” because it’s safer for us to patch
m

Maximilian Schulz

04/28/2023, 7:48 AM
@Zanie in the context of this thread I was more referring to the fact of my post that when I added my own logging handler in a package it was somehow overwritten and not called in the context of a flow or task - in the a normal python execution context it was called - and imho that shouldn’t happen 🙂
regarding the actual problem with
sys.stdout
I believe we would profit a lot if it was at least an option, but have to dig deeper into that - as many tools provide some form of
stdout
binding (e.g. subprocesses) it would be vital to us so we can use python packages as they come - I am thinking something like this as an option: https://stackoverflow.com/questions/19425736/how-to-redirect-stdout-and-stderr-to-logger-in-python/31688396#31688396
z

Zanie

04/28/2023, 2:15 PM
Unfortunately Python logging doesn't really support incremental updates well and we need to take control of logging to provide the UX our users want. We provide full control over the logging configuration so you can usually do what you need. If you want to dynamically create handlers support is complicated, if you open a GitHub issue we can try.
We could support stdout but it'd probably need to be a global setting rather than a task / flow option because it's harder to constrain to a specific scope in the way we do for log prints.
m

Maximilian Schulz

04/28/2023, 3:29 PM
Will look deeper into this, thanks @Zanie for your responses!