https://prefect.io
Join Slack
Hi does anyone have an example on using the <extra loggers>? I have a snippet in `custom.py` like th...
# ask-community
m
Hi does anyone have an example on using the extra loggers? I have a snippet in 
custom.py
like this
Copy code
import logging
import sys

def run():
  logger = logging.getLogger("custom")
  logger.setLevel('INFO')
  log_stream = logging.StreamHandler(sys.stdout)
  log_stream.setFormatter(LOG_FORMAT)
  logger.addHandler(log_stream)
and also have
Copy code
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
in the toml config, but logs from the custom logger don’t seem to be captured by Prefect and show up in the cloud console.
k
Can I see how you use it in the Flow? Are you on a DaskExecutor?
m
For example, in 
flow.py
, i have this
Copy code
import custom

@task
def run():
  custom.run()
No, I am not on a DaskExecutor
k
Ok let me try to make an example
m
Any luck?
k
Wait sorry juggling a couple of things
Working on this now
What you have seems to work:
Copy code
#test.py
import logging
import sys

logger = logging.getLogger("custom")
logger.setLevel('DEBUG')

def run():
    <http://logger.info|logger.info>('This is an info message')
    logger.debug('This is an debug message')
and the flow:
Copy code
from prefect import Flow, task
from test import run

@task
def abc():
    run()
    return

with Flow("...") as flow:
    abc()

flow.run()
and my config.toml
Copy code
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
and my logs:
Copy code
(prefect) kevinkho@Kevins-MacBook-Pro sources % python query.py
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Beginning Flow run for '...'
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Starting task run...
[2022-02-24 14:51:57-0500] INFO - custom | This is an info message
[2022-02-24 14:51:57-0500] DEBUG - custom | This is an debug message
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Finished task run for task with final state: 'Success'
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
Without the config.toml entry it is just:
Copy code
[2022-02-24 15:00:03-0500] INFO - prefect.FlowRunner | Beginning Flow run for '...'
[2022-02-24 15:00:03-0500] INFO - prefect.TaskRunner | Task 'abc': Starting task run...
[2022-02-24 15:00:03-0500] INFO - prefect.TaskRunner | Task 'abc': Finished task run for task with final state: 'Success'
[2022-02-24 15:00:03-0500] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
m
hmm may I ask when this feature was introduced?
k
i joined Prefect around 0.14.4 and I feel it was already there. Are you before that?
it was fixed in 0.9.6
m
oh i found it
yeah 0.9.6
We are on 0.14.11 and using a custom agent with 
run_config
.Could you point me to the code where it is implemented for other types of agents? I wonder if we need to implement this passdown logic ourselves in our own custom agent.
k
So you can do it in the RunConfig with:
Copy code
flow.run_config = RunConfig(env={"PREFECT__LOGGING__EXTRA_LOGGERS":"['custom']")
and that should work for all agents (at least for the ones we have off the shelf) This pass down logic should happen when you merge the env from the RunConfig
You can trace this line for the RunConfig enviroment pass down for the Docker Agent for example
m
If I pass it in as an env to 
run_config
, it does work, but it doesn’t if I put it in 
config.toml
as the doc suggests:
Copy code
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
as this is out of side 
run_config
and doesn’t get picked up?
Would you mind clarifying how you supply 
config.toml
?
is this an agent level, flow level or flow run level setting?
k
I think you can pass the env on the agent side with 
prefect … agent start --env PREFECT___LOGGING…
._ The 
config.toml
is parsed when Prefect is imported so if you have a Flow with a container, it would have to go in the container
By putting the env on the agent, it gets passed to the flow it deploys
m
I specify the extra loggers in my user level 
config.toml
, which takes precedence over the default in the container
but the flow doesn’t seem to pick it up?
the run configs do get picked up from this user level config.toml though
k
What is your agent setup? Is it a container based?
m
yes it is
k
You already have the 
config.toml
inside the container and it doesn’t get picked up?
m
the agent itself runs in a container but all the flow runs are running in a container as well
i meant i already have the config.toml in the flow runs’ containers
k
Let me read some code to be sure where it gets picked up one sec
m
i don’t have a user level config.toml override in the agent container
sure thanks Kevin
k
Can’t immediate find if it’s the agent 
config.toml
or container 
config.toml
that carries it over. Will test later tonight
👍 1
m
the reason we would like to put this in the flow run container is that different flows can use different loggers and this way gives us the maximum flexibility
k
I can replicate that 
config.toml
inside the container did not work but RunConfig env variable work. I’ll need to ask some people on the team what the intended behavior is
m
No problem, please keep me posted Kevin!
@Kevin Kho another related issue, it seems that if I do 
import prefect.config
in the flow, the user configs I provide is not available either, which kind of contradicts what it says at https://docs.prefect.io/core/concepts/configuration.html?
k
Team members just got back to me and 
config.toml
only works for local agent. You need to use the environment variable for the container based Flows. About this let me test quickly
m
i guess it is probably due to the same reason
k
So yeah I do see it on local with:
Copy code
from prefect import config

print(dict(config))
but don’t know about Flows with backend in a container readily
m
if it is not mentioned, could you help update the docs to make it clear? It took us a day to find out in this hard way, only with your help.
Also, i wonder if there is a reason not allowing flows to use config.toml? Can it be made happen?
k
Dunno yet, will open a ticket. Because I think we are also moving away from 
config.toml
for Prefect 2.0. Will open an issue later to track
m
oh that’s interesting to know
what will come in place of that then?
k
It seems the feature hasn’t been announced yet but will be in like 2 weeks time I think
m
looking forward to it!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by