Milton
02/24/2022, 7:19 PMcustom.py
like this
import logging
import sys
def run():
logger = logging.getLogger("custom")
logger.setLevel('INFO')
log_stream = logging.StreamHandler(sys.stdout)
log_stream.setFormatter(LOG_FORMAT)
logger.addHandler(log_stream)
and also have
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
in the toml config, but logs from the custom logger don’t seem to be captured by Prefect and show up in the cloud console.Kevin Kho
Milton
02/24/2022, 7:23 PMflow.py
, i have this
import custom
@task
def run():
custom.run()
Kevin Kho
Milton
02/24/2022, 7:45 PMKevin Kho
#test.py
import logging
import sys
logger = logging.getLogger("custom")
logger.setLevel('DEBUG')
def run():
<http://logger.info|logger.info>('This is an info message')
logger.debug('This is an debug message')
and the flow:
from prefect import Flow, task
from test import run
@task
def abc():
run()
return
with Flow("...") as flow:
abc()
flow.run()
and my config.toml
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
and my logs:
(prefect) kevinkho@Kevins-MacBook-Pro sources % python query.py
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Beginning Flow run for '...'
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Starting task run...
[2022-02-24 14:51:57-0500] INFO - custom | This is an info message
[2022-02-24 14:51:57-0500] DEBUG - custom | This is an debug message
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Finished task run for task with final state: 'Success'
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
[2022-02-24 15:00:03-0500] INFO - prefect.FlowRunner | Beginning Flow run for '...'
[2022-02-24 15:00:03-0500] INFO - prefect.TaskRunner | Task 'abc': Starting task run...
[2022-02-24 15:00:03-0500] INFO - prefect.TaskRunner | Task 'abc': Finished task run for task with final state: 'Success'
[2022-02-24 15:00:03-0500] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
Milton
02/24/2022, 8:07 PMKevin Kho
Milton
02/24/2022, 8:10 PMrun_config
.Could you point me to the code where it is implemented for other types of agents? I wonder if we need to implement this passdown logic ourselves in our own custom agent.Kevin Kho
flow.run_config = RunConfig(env={"PREFECT__LOGGING__EXTRA_LOGGERS":"['custom']")
and that should work for all agents (at least for the ones we have off the shelf)
This pass down logic should happen when you merge the env from the RunConfigMilton
02/24/2022, 10:37 PMrun_config
, it does work, but it doesn’t if I put it in config.toml
as the doc suggests:
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
as this is out of side run_config
and doesn’t get picked up?config.toml
?Kevin Kho
prefect … agent start --env PREFECT___LOGGING…
._ The config.toml
is parsed when Prefect is imported so if you have a Flow with a container, it would have to go in the containerMilton
02/24/2022, 10:53 PMconfig.toml
, which takes precedence over the default in the containerKevin Kho
Milton
02/24/2022, 11:02 PMKevin Kho
config.toml
inside the container and it doesn’t get picked up?Milton
02/24/2022, 11:05 PMKevin Kho
Milton
02/24/2022, 11:06 PMKevin Kho
config.toml
or container config.toml
that carries it over. Will test later tonightMilton
02/24/2022, 11:21 PMKevin Kho
config.toml
inside the container did not work but RunConfig env variable work. I’ll need to ask some people on the team what the intended behavior isMilton
02/25/2022, 2:41 PMimport prefect.config
in the flow, the user configs I provide is not available either, which kind of contradicts what it says at https://docs.prefect.io/core/concepts/configuration.html?Kevin Kho
config.toml
only works for local agent. You need to use the environment variable for the container based Flows. About this let me test quicklyMilton
02/25/2022, 6:42 PMKevin Kho
from prefect import config
print(dict(config))
but don’t know about Flows with backend in a container readilyMilton
02/25/2022, 6:52 PMKevin Kho
config.toml
for Prefect 2.0. Will open an issue later to trackMilton
02/25/2022, 7:16 PMKevin Kho
Milton
02/25/2022, 7:30 PM