https://prefect.io
Join Slack
when i run flows locally, my logging works fine, but if i run flows in prefect cloud, i get zero log...
# ask-community
z
when i run flows locally, my logging works fine, but if i run flows in prefect cloud, i get zero logging output. am i doing something blaringly wrong?
im doing this in my task entrypoints, (django app settings files), where LOGGING is a dict of log config
Copy code
# set the django loggers to inherit from prefect config
_loggers = list(LOGGING["loggers"])
PREFECT_EXTRA_LOGGERS = str(_loggers)
os.environ["PREFECT__LOGGING__EXTRA_LOGGERS"] = PREFECT_EXTRA_LOGGERS
k
You don’t even get the Prefect logs on Cloud?
z
sorry i shouldve been more specific. I don't get any logs that aren't logged with the logger from prefect.utitilites.logging
so i get builtin logs and logs that the app generates itself
but if i did
Copy code
logger = logging.getLogger(__name__)
<http://logger.info|logger.info>("sup)
i'd get it fine locally but would never see it in cloud
k
This looks right. What happens when you run your flow with 
--show-flow-logs
on a local agent?
z
i don't even have to do that, it just works locally
k
I’m wondering because I’ve seen cases with 
dask-jobqueue
specifically where the logger works locally but doesn’t on Cloud. 
--show-flow-logs
will show it though. This happens when the extra logger is a different process than the one Prefect is running on and doesn’t get the CloudHandler attached to it. Do you think that could be the case here?
z
could be? we use a k8s prefect agent and just the default executor fwiw
i'm wondering if maybe i'm setting that env variable too late?
can you link me to where that is used when running via the k8s agent in prefect cloud?
k
Let me dig a bit
Team said the local run showing all the logs is not indication that the extra loggers are configured correctly. The place this is set is here. So it pulls from config when called and attaches the CloudHandler. I guess what you can do is check if they make it in 
prefect.context.config
correctly?
z
is it wild to call configure_extra_loggers explicitly in my entrypoint?
or does that not do anything unless in the context of a flow
k
Yeah exactly. It needs to be in the Flow.
z
hm ok. not really clear what is going on but i guess i will keep digging
k
Log the config value to see if the EXTRA_LOGGERS made it correctly?
z
looks like its not
where does it read in that env var?
im going to try just setting that value explicitly on context
and see if that does it
k
Oh it won’t work just by setting it. This happens during the import. You can pass in the ENV VAR through the RunConfig or spinning up the agent. It also may work if it isn’t the entrypoint of the container?
z
yeah it didn't work lol.. yeah i tried the env var in the run config and on the container and in the k8s pod itself
nothing seems to work
k
Could it be that those logs are DEBUG level?
z
nope, they are all set to info
k
Could you show the RunConfig snippet?
z
Copy code
PREFECT_JOB_ENVIRONMENT = {
    "ENVIRONMENT": ENVIRONMENT,
    "DJANGO_SETTINGS_MODULE": "sbdeco.config.settings.production",
    "PREFECT__LOGGING__EXTRA_LOGGERS": PREFECT_EXTRA_LOGGERS,
}
PREFECT_RUN_CONFIG = KubernetesRun(labels=[PREFECT_AGENT_LABEL], image=PREFECT_IMAGE, env=PREFECT_JOB_ENVIRONMENT)
k
Using the RunConfig doesn’t make them show up when you log the config?
Are these standard packages? Like boto3? or are they custom modules?
z
right, its just an empty array
internal packages, standard packages, and modules in the same repo
none of them log anything
c
@Zach Schumacher - I think I dealt with this exact problem today, see the docs here. Based on your code snippet above it looks like you're saying that in your code, you have 
import logging
then create your logger as 
logger = logging.getLogger(__name__)
. However to have the logs work successfully in prefect, you need 
import prefect
, 
logger = prefect.context.get("logger")
. Note that you can't use 
from prefect import context
and that you invoke the logger WITHIN each task where you want logging, both per the docs.
Hope this helps!
z
That’s okay if I need to log from a task, but not when I’m logging from other packages that don’t have prefect as a dep
c
Ahh got it, nvm!
9 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by