https://prefect.io logo
m

Milton

02/24/2022, 7:19 PM
Hi does anyone have an example on using the extra loggers? I have a snippet in
custom.py
like this
Copy code
import logging
import sys

def run():
  logger = logging.getLogger("custom")
  logger.setLevel('INFO')
  log_stream = logging.StreamHandler(sys.stdout)
  log_stream.setFormatter(LOG_FORMAT)
  logger.addHandler(log_stream)
and also have
Copy code
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
in the toml config, but logs from the custom logger don’t seem to be captured by Prefect and show up in the cloud console.
k

Kevin Kho

02/24/2022, 7:20 PM
Can I see how you use it in the Flow? Are you on a DaskExecutor?
m

Milton

02/24/2022, 7:23 PM
For example, in
flow.py
, i have this
Copy code
import custom

@task
def run():
  custom.run()
No, I am not on a DaskExecutor
k

Kevin Kho

02/24/2022, 7:25 PM
Ok let me try to make an example
m

Milton

02/24/2022, 7:45 PM
Any luck?
k

Kevin Kho

02/24/2022, 7:46 PM
Wait sorry juggling a couple of things
Working on this now
What you have seems to work:
Copy code
#test.py
import logging
import sys

logger = logging.getLogger("custom")
logger.setLevel('DEBUG')

def run():
    <http://logger.info|logger.info>('This is an info message')
    logger.debug('This is an debug message')
and the flow:
Copy code
from prefect import Flow, task
from test import run

@task
def abc():
    run()
    return

with Flow("...") as flow:
    abc()

flow.run()
and my config.toml
Copy code
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
and my logs:
Copy code
(prefect) kevinkho@Kevins-MacBook-Pro sources % python query.py
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Beginning Flow run for '...'
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Starting task run...
[2022-02-24 14:51:57-0500] INFO - custom | This is an info message
[2022-02-24 14:51:57-0500] DEBUG - custom | This is an debug message
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Finished task run for task with final state: 'Success'
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
Without the config.toml entry it is just:
Copy code
[2022-02-24 15:00:03-0500] INFO - prefect.FlowRunner | Beginning Flow run for '...'
[2022-02-24 15:00:03-0500] INFO - prefect.TaskRunner | Task 'abc': Starting task run...
[2022-02-24 15:00:03-0500] INFO - prefect.TaskRunner | Task 'abc': Finished task run for task with final state: 'Success'
[2022-02-24 15:00:03-0500] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
m

Milton

02/24/2022, 8:07 PM
hmm may I ask when this feature was introduced?
k

Kevin Kho

02/24/2022, 8:09 PM
i joined Prefect around 0.14.4 and I feel it was already there. Are you before that?
it was fixed in 0.9.6
m

Milton

02/24/2022, 8:10 PM
oh i found it
yeah 0.9.6
We are on 0.14.11 and using a custom agent with
run_config
.Could you point me to the code where it is implemented for other types of agents? I wonder if we need to implement this passdown logic ourselves in our own custom agent.
k

Kevin Kho

02/24/2022, 8:53 PM
So you can do it in the RunConfig with:
Copy code
flow.run_config = RunConfig(env={"PREFECT__LOGGING__EXTRA_LOGGERS":"['custom']")
and that should work for all agents (at least for the ones we have off the shelf) This pass down logic should happen when you merge the env from the RunConfig
You can trace this line for the RunConfig enviroment pass down for the Docker Agent for example
m

Milton

02/24/2022, 10:37 PM
If I pass it in as an env to
run_config
, it does work, but it doesn’t if I put it in
config.toml
as the doc suggests:
Copy code
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
as this is out of side
run_config
and doesn’t get picked up?
Would you mind clarifying how you supply
config.toml
?
is this an agent level, flow level or flow run level setting?
k

Kevin Kho

02/24/2022, 10:47 PM
I think you can pass the env on the agent side with
prefect … agent start --env PREFECT___LOGGING…
._ The
config.toml
is parsed when Prefect is imported so if you have a Flow with a container, it would have to go in the container
By putting the env on the agent, it gets passed to the flow it deploys
m

Milton

02/24/2022, 10:53 PM
I specify the extra loggers in my user level
config.toml
, which takes precedence over the default in the container
but the flow doesn’t seem to pick it up?
the run configs do get picked up from this user level config.toml though
k

Kevin Kho

02/24/2022, 10:58 PM
What is your agent setup? Is it a container based?
m

Milton

02/24/2022, 11:02 PM
yes it is
k

Kevin Kho

02/24/2022, 11:04 PM
You already have the
config.toml
inside the container and it doesn’t get picked up?
m

Milton

02/24/2022, 11:05 PM
the agent itself runs in a container but all the flow runs are running in a container as well
i meant i already have the config.toml in the flow runs’ containers
k

Kevin Kho

02/24/2022, 11:06 PM
Let me read some code to be sure where it gets picked up one sec
m

Milton

02/24/2022, 11:06 PM
i don’t have a user level config.toml override in the agent container
sure thanks Kevin
k

Kevin Kho

02/24/2022, 11:17 PM
Can’t immediate find if it’s the agent
config.toml
or container
config.toml
that carries it over. Will test later tonight
👍 1
m

Milton

02/24/2022, 11:21 PM
the reason we would like to put this in the flow run container is that different flows can use different loggers and this way gives us the maximum flexibility
k

Kevin Kho

02/25/2022, 5:35 AM
I can replicate that
config.toml
inside the container did not work but RunConfig env variable work. I’ll need to ask some people on the team what the intended behavior is
m

Milton

02/25/2022, 2:41 PM
No problem, please keep me posted Kevin!
@Kevin Kho another related issue, it seems that if I do
import prefect.config
in the flow, the user configs I provide is not available either, which kind of contradicts what it says at https://docs.prefect.io/core/concepts/configuration.html?
k

Kevin Kho

02/25/2022, 6:42 PM
Team members just got back to me and
config.toml
only works for local agent. You need to use the environment variable for the container based Flows. About this let me test quickly
m

Milton

02/25/2022, 6:42 PM
i guess it is probably due to the same reason
k

Kevin Kho

02/25/2022, 6:44 PM
So yeah I do see it on local with:
Copy code
from prefect import config

print(dict(config))
but don’t know about Flows with backend in a container readily
m

Milton

02/25/2022, 6:52 PM
if it is not mentioned, could you help update the docs to make it clear? It took us a day to find out in this hard way, only with your help.
Also, i wonder if there is a reason not allowing flows to use config.toml? Can it be made happen?
k

Kevin Kho

02/25/2022, 7:02 PM
Dunno yet, will open a ticket. Because I think we are also moving away from
config.toml
for Prefect 2.0. Will open an issue later to track
m

Milton

02/25/2022, 7:16 PM
oh that’s interesting to know
what will come in place of that then?
k

Kevin Kho

02/25/2022, 7:25 PM
It seems the feature hasn’t been announced yet but will be in like 2 weeks time I think
m

Milton

02/25/2022, 7:30 PM
looking forward to it!
2 Views