Thread
#prefect-community
    Zach Schumacher

    Zach Schumacher

    1 year ago
    when i run flows locally, my logging works fine, but if i run flows in prefect cloud, i get zero logging output. am i doing something blaringly wrong?
    im doing this in my task entrypoints, (django app settings files), where LOGGING is a dict of log config
    # set the django loggers to inherit from prefect config
    _loggers = list(LOGGING["loggers"])
    PREFECT_EXTRA_LOGGERS = str(_loggers)
    os.environ["PREFECT__LOGGING__EXTRA_LOGGERS"] = PREFECT_EXTRA_LOGGERS
    Kevin Kho

    Kevin Kho

    1 year ago
    You don’t even get the Prefect logs on Cloud?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    sorry i shouldve been more specific. I don't get any logs that aren't logged with the logger from prefect.utitilites.logging
    so i get builtin logs and logs that the app generates itself
    but if i did
    logger = logging.getLogger(__name__)
    <http://logger.info|logger.info>("sup)
    i'd get it fine locally but would never see it in cloud
    Kevin Kho

    Kevin Kho

    1 year ago
    This looks right. What happens when you run your flow with
    --show-flow-logs
    on a local agent?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    i don't even have to do that, it just works locally
    Kevin Kho

    Kevin Kho

    1 year ago
    I’m wondering because I’ve seen cases with
    dask-jobqueue
    specifically where the logger works locally but doesn’t on Cloud.
    --show-flow-logs
    will show it though. This happens when the extra logger is a different process than the one Prefect is running on and doesn’t get the CloudHandler attached to it. Do you think that could be the case here?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    could be? we use a k8s prefect agent and just the default executor fwiw
    i'm wondering if maybe i'm setting that env variable too late?
    can you link me to where that is used when running via the k8s agent in prefect cloud?
    Kevin Kho

    Kevin Kho

    1 year ago
    Let me dig a bit
    Team said the local run showing all the logs is not indication that the extra loggers are configured correctly. The place this is set is here. So it pulls from config when called and attaches the CloudHandler. I guess what you can do is check if they make it in
    prefect.context.config
    correctly?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    is it wild to call configure_extra_loggers explicitly in my entrypoint?
    or does that not do anything unless in the context of a flow
    Kevin Kho

    Kevin Kho

    1 year ago
    Yeah exactly. It needs to be in the Flow.
    Zach Schumacher

    Zach Schumacher

    1 year ago
    hm ok. not really clear what is going on but i guess i will keep digging
    Kevin Kho

    Kevin Kho

    1 year ago
    Log the config value to see if the EXTRA_LOGGERS made it correctly?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    looks like its not
    where does it read in that env var?
    im going to try just setting that value explicitly on context
    and see if that does it
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh it won’t work just by setting it. This happens during the import. You can pass in the ENV VAR through the RunConfig or spinning up the agent. It also may work if it isn’t the entrypoint of the container?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    yeah it didn't work lol.. yeah i tried the env var in the run config and on the container and in the k8s pod itself
    nothing seems to work
    Kevin Kho

    Kevin Kho

    1 year ago
    Could it be that those logs are DEBUG level?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    nope, they are all set to info
    Kevin Kho

    Kevin Kho

    1 year ago
    Could you show the RunConfig snippet?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    PREFECT_JOB_ENVIRONMENT = {
        "ENVIRONMENT": ENVIRONMENT,
        "DJANGO_SETTINGS_MODULE": "sbdeco.config.settings.production",
        "PREFECT__LOGGING__EXTRA_LOGGERS": PREFECT_EXTRA_LOGGERS,
    }
    PREFECT_RUN_CONFIG = KubernetesRun(labels=[PREFECT_AGENT_LABEL], image=PREFECT_IMAGE, env=PREFECT_JOB_ENVIRONMENT)
    Kevin Kho

    Kevin Kho

    1 year ago
    Using the RunConfig doesn’t make them show up when you log the config?
    Are these standard packages? Like boto3? or are they custom modules?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    right, its just an empty array
    internal packages, standard packages, and modules in the same repo
    none of them log anything
    Claire Herdeman

    Claire Herdeman

    1 year ago
    @Zach Schumacher - I think I dealt with this exact problem today, see the docs here. Based on your code snippet above it looks like you're saying that in your code, you have
    import logging
    then create your logger as
    logger = logging.getLogger(__name__)
    . However to have the logs work successfully in prefect, you need
    import prefect
    ,
    logger = prefect.context.get("logger")
    . Note that you can't use
    from prefect import context
    and that you invoke the logger WITHIN each task where you want logging, both per the docs.
    Hope this helps!
    Zach Schumacher

    Zach Schumacher

    1 year ago
    That’s okay if I need to log from a task, but not when I’m logging from other packages that don’t have prefect as a dep
    Claire Herdeman

    Claire Herdeman

    1 year ago
    Ahh got it, nvm!