Has the behaviour of `PREFECT LOGGING EXTRA LOGGERS` changed Prefect Community #prefect-community

Has the behaviour of `PREFECT_LOGGING_EXTRA_LOGGER...

Tim Galvin

12/03/2022, 8:18 AM

Has the behaviour of

PREFECT_LOGGING_EXTRA_LOGGERS

changed from

2.6.7

2.6.9

at all? I was previously using it successfully in combination with

DaskTaskExecutor

and

dask_jobqueue.SLURMCluster

to capture logs from another package. It seems though with an updated to

2.6.8

2.6.9

I have lost this ability. I do see the logs being printed to my slurm output files, and I do see it is formatted in the

prefect

style, but I am not seeing these logs being submitted through to my self-hosted Orion server (as presented by the web UI). Any ideas?

Tim Galvin

12/03/2022, 9:37 AM

Ahhh I see that there is now a

logging.yml

capability. I suppose this is now changing the behaviour of the

PREFECT_LOGGING_EXTRA_LOGGERS

variable I was using. Can anyone confirm?

Tim Galvin

12/03/2022, 9:50 AM

Lets say in an earlier version of

prefect

I would use

PREFECT_LOGGING_EXTRA_LOGGERS="mymodule"

, what would be the proper way of using the new interface to capture these logs? I see in the docs I can do something like

Copy code

PREFECT_LOGGING_[PATH]_[TO]_[KEY]=value

Is there a way of specifying these extra modules? Would it be something like

Copy code

export PREFECT_LOGGING_LOGGERS_MYMODULE_LEVEL=INFO
export PREFECT_LOGGING_LOGGER_MYMODULE_HANDLERS="[orion, console]"

Tim Galvin

12/03/2022, 10:35 AM

Actually, now I am not sure. Reading the code in

prefect/src/logging/configuration.py

seems to have the

PREFECT_LOGGING_EXTRA_LOGGERS

logic in there, and as far as I can tell from the

logging.yml

file these extra modules should be logged to both the

orion

and

console

handlers. But it seems like mine are only recorded to the console.

Mason Menges

12/05/2022, 7:05 PM

Hey Tim, the only thing I know of that was included in 2.6.9 was Dask Worker logs should be getting surfaced on the task run logs within prefect now, if have a Minimum Reproducible example of the code/setup you're running it might be worth opening a bug on the github repo. Missed the new message you posted, you can ignore this haha 😅

4 Views

Open in Slack

Previous Next