https://prefect.io logo
Join Slack
Powered by
Hi - I’d like some advice on having logs show up i...
# ask-community
a
Hi - I’d like some advice on having logs show up in Prefect Cloud as well as streamed locally to stderr/stdout (the latter being currently the case). I currently have some code that is wrapped in tasks/flows which produces these logs (and has an associated 
logger
). I was hoping that I could perhaps add this as an extra logger to Prefect (or add Prefect’s logger as one of the handlers), resulting in getting logs in cloud and on local stderr. Has anyone set up something similar to this before? (I tried naively adding the package logger to Prefect’s 
PREFECT_LOGGING_EXTRA_LOGGERS
env var, but this doesn’t seem to have worked as expected).
l
Hi! I would have expected adding it to extra loggers to address your issue. Fundamentally what I think you want to happen is have the CloudHandler attached to your custom logger, which is what that extra loggers config is supposed to do for you (https://github.com/PrefectHQ/prefect/blob/e8e29e173948777d5f67295fc18e7d7475bc9671/src/prefect/utilities/logging.py#L209)
a
hi @Laura Lorenz (she/her) - yes, this is what I was about to start looking at
l
Do you mind sending me exactly what your env var looks like?
a
not sure why the 
EXTRA_LOGGERS
env var isn’t doing this for me
yeah sure
Copy code
echo $PREFECT__LOGGING__EXTRA_LOGGERS                                                          
['worker.integrations.dealroom.downloader']
this is where there is some package/module structure like 
worker/integrations/dealroom/downloader.py
and a logger configured as 
logger = logging.getLogger(__name__)
I was thinking about just using 
get_logger
from 
prefect.utilities.logging
and adding that as a handler to the logger in that module
or just grab the 
CloudHandler
directly as you say
btw - it’s also possible I’m testing this incorrectly - locally I’m just running the flow and was expecting to see these logs appear in the local Prefect log stream. Maybe that’s not right
l
Yes, I think if extra loggers isn’t working for some reason (which I’m going to try and reproduce) adding the CloudHandler directly is an option. Yes, if the extra loggers config was working correctly I believe you would see them in the local Prefect stdout log stream, because it would attach both a CloudHandler and a StreamHandler with your prefect config at the same time.
a
yeah exactly
cool - I think I’ll wait to see if you can repro
👍 1
the other part of this btw is that it would be really cool if I could add a package level logger and have everything under that package write to Prefect Stream/Cloud
i.e. adding 
worker.integrations
and getting all loggers under that
to avoid having to add a parameter for each package under 
integrations.<package>.py
j
There was a bug in the parsing of the environment variable
Copy code
PREFECT__LOGGING__EXTRA_LOGGERS
that was patched in 
0.9.6
. If you upgrade your Prefect version and set it as a string representation of a list then it should be parsed properly! e.g.
Copy code
export 
PREFECT__LOGGING__EXTRA_LOGGERS=                                                        "['your_extra_logging_package']"
a
awesome! That seems to be working.
l
Yay!
a
and as expected/would make sense, it’s fine to specify a logger further up the package hierarchy
great
let me deploy that and see how we do
👍 1
thanks for your help - appreciate it!
@Kostas Chalikias - this was the last thread on logging
3 Views
Open in Slack
PreviousNext
Powered by