a

    Arlo Bryer

    2 years ago
    Hi - I’d like some advice on having logs show up in Prefect Cloud as well as streamed locally to stderr/stdout (the latter being currently the case). I currently have some code that is wrapped in tasks/flows which produces these logs (and has an associated
    logger
    ). I was hoping that I could perhaps add this as an extra logger to Prefect (or add Prefect’s logger as one of the handlers), resulting in getting logs in cloud and on local stderr. Has anyone set up something similar to this before? (I tried naively adding the package logger to Prefect’s
    PREFECT_LOGGING_EXTRA_LOGGERS
    env var, but this doesn’t seem to have worked as expected).
    Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    Hi! I would have expected adding it to extra loggers to address your issue. Fundamentally what I think you want to happen is have the CloudHandler attached to your custom logger, which is what that extra loggers config is supposed to do for you (https://github.com/PrefectHQ/prefect/blob/e8e29e173948777d5f67295fc18e7d7475bc9671/src/prefect/utilities/logging.py#L209)
    a

    Arlo Bryer

    2 years ago
    hi @Laura Lorenz (she/her) - yes, this is what I was about to start looking at
    Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    Do you mind sending me exactly what your env var looks like?
    a

    Arlo Bryer

    2 years ago
    not sure why the
    EXTRA_LOGGERS
    env var isn’t doing this for me
    yeah sure
    echo $PREFECT__LOGGING__EXTRA_LOGGERS                                                          
    ['worker.integrations.dealroom.downloader']
    this is where there is some package/module structure like
    worker/integrations/dealroom/downloader.py
    and a logger configured as
    logger = logging.getLogger(__name__)
    I was thinking about just using
    get_logger
    from
    prefect.utilities.logging
    and adding that as a handler to the logger in that module
    or just grab the
    CloudHandler
    directly as you say
    btw - it’s also possible I’m testing this incorrectly - locally I’m just running the flow and was expecting to see these logs appear in the local Prefect log stream. Maybe that’s not right
    Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    Yes, I think if extra loggers isn’t working for some reason (which I’m going to try and reproduce) adding the CloudHandler directly is an option. Yes, if the extra loggers config was working correctly I believe you would see them in the local Prefect stdout log stream, because it would attach both a CloudHandler and a StreamHandler with your prefect config at the same time.
    a

    Arlo Bryer

    2 years ago
    yeah exactly
    cool - I think I’ll wait to see if you can repro
    the other part of this btw is that it would be really cool if I could add a package level logger and have everything under that package write to Prefect Stream/Cloud
    i.e. adding
    worker.integrations
    and getting all loggers under that
    to avoid having to add a parameter for each package under
    integrations.<package>.py
    j

    josh

    2 years ago
    There was a bug in the parsing of the environment variable
    PREFECT__LOGGING__EXTRA_LOGGERS
    that was patched in
    0.9.6
    . If you upgrade your Prefect version and set it as a string representation of a list then it should be parsed properly! e.g.
    export 
    PREFECT__LOGGING__EXTRA_LOGGERS=                                                        "['your_extra_logging_package']"
    a

    Arlo Bryer

    2 years ago
    awesome! That seems to be working.
    Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    Yay!
    a

    Arlo Bryer

    2 years ago
    and as expected/would make sense, it’s fine to specify a logger further up the package hierarchy
    great
    let me deploy that and see how we do
    thanks for your help - appreciate it!
    @Kostas Chalikias - this was the last thread on logging