https://prefect.io logo
Title
a

Arlo Bryer

03/05/2020, 5:06 PM
Hi - I’d like some advice on having logs show up in Prefect Cloud as well as streamed locally to stderr/stdout (the latter being currently the case). I currently have some code that is wrapped in tasks/flows which produces these logs (and has an associated
logger
). I was hoping that I could perhaps add this as an extra logger to Prefect (or add Prefect’s logger as one of the handlers), resulting in getting logs in cloud and on local stderr. Has anyone set up something similar to this before? (I tried naively adding the package logger to Prefect’s
PREFECT_LOGGING_EXTRA_LOGGERS
env var, but this doesn’t seem to have worked as expected).
l

Laura Lorenz (she/her)

03/05/2020, 5:12 PM
Hi! I would have expected adding it to extra loggers to address your issue. Fundamentally what I think you want to happen is have the CloudHandler attached to your custom logger, which is what that extra loggers config is supposed to do for you (https://github.com/PrefectHQ/prefect/blob/e8e29e173948777d5f67295fc18e7d7475bc9671/src/prefect/utilities/logging.py#L209)
a

Arlo Bryer

03/05/2020, 5:14 PM
hi @Laura Lorenz (she/her) - yes, this is what I was about to start looking at
l

Laura Lorenz (she/her)

03/05/2020, 5:14 PM
Do you mind sending me exactly what your env var looks like?
a

Arlo Bryer

03/05/2020, 5:14 PM
not sure why the
EXTRA_LOGGERS
env var isn’t doing this for me
yeah sure
echo $PREFECT__LOGGING__EXTRA_LOGGERS                                                          
['worker.integrations.dealroom.downloader']
this is where there is some package/module structure like
worker/integrations/dealroom/downloader.py
and a logger configured as
logger = logging.getLogger(__name__)
I was thinking about just using
get_logger
from
prefect.utilities.logging
and adding that as a handler to the logger in that module
or just grab the
CloudHandler
directly as you say
btw - it’s also possible I’m testing this incorrectly - locally I’m just running the flow and was expecting to see these logs appear in the local Prefect log stream. Maybe that’s not right
l

Laura Lorenz (she/her)

03/05/2020, 5:21 PM
Yes, I think if extra loggers isn’t working for some reason (which I’m going to try and reproduce) adding the CloudHandler directly is an option. Yes, if the extra loggers config was working correctly I believe you would see them in the local Prefect stdout log stream, because it would attach both a CloudHandler and a StreamHandler with your prefect config at the same time.
a

Arlo Bryer

03/05/2020, 5:21 PM
yeah exactly
cool - I think I’ll wait to see if you can repro
👍 1
the other part of this btw is that it would be really cool if I could add a package level logger and have everything under that package write to Prefect Stream/Cloud
i.e. adding
worker.integrations
and getting all loggers under that
to avoid having to add a parameter for each package under
integrations.<package>.py
j

josh

03/05/2020, 5:24 PM
There was a bug in the parsing of the environment variable
PREFECT__LOGGING__EXTRA_LOGGERS
that was patched in
0.9.6
. If you upgrade your Prefect version and set it as a string representation of a list then it should be parsed properly! e.g.
export 
PREFECT__LOGGING__EXTRA_LOGGERS=                                                        "['your_extra_logging_package']"
a

Arlo Bryer

03/05/2020, 5:27 PM
awesome! That seems to be working.
l

Laura Lorenz (she/her)

03/05/2020, 5:27 PM
Yay!
a

Arlo Bryer

03/05/2020, 5:27 PM
and as expected/would make sense, it’s fine to specify a logger further up the package hierarchy
great
let me deploy that and see how we do
👍 1
thanks for your help - appreciate it!
@Kostas Chalikias - this was the last thread on logging