Hi does anyone have an example on using the

Question

Hi does anyone have an example on using the <https docs prefect io core concepts logging html extra loggers|extra loggers> I have a snippet in `custom py` like this ```import logging import sys def run logger = logging getLogger custom logger setLevel INFO log stream = logging StreamHandler sys stdout log stream setFormatter LOG FORMAT logger addHandler log stream ``` and also have ``` logging Extra loggers for Prefect log configuration extra loggers = custom ``` in the toml config but logs from the custom logger don t seem to be captured by Prefect and show up in the cloud console

Kevin Kho · Answer

Can I see how you use it in the Flow Are you on a DaskExecutor

Milton · Answer

For example in `flow py` i have this ```import custom task def run custom run ```

Milton · Answer

No I am not on a DaskExecutor

Kevin Kho · Answer

Ok let me try to make an example

Milton · Answer

Any luck

Kevin Kho · Answer

Wait sorry juggling a couple of things

Kevin Kho · Answer

Working on this now

Kevin Kho · Answer

What you have seems to work ``` test py import logging import sys logger = logging getLogger custom logger setLevel DEBUG def run <http logger info|logger info> This is an info message logger debug This is an debug message ``` and the flow ```from prefect import Flow task from test import run task def abc run return with Flow as flow abc flow run ``` and my config toml ``` logging Extra loggers for Prefect log configuration extra loggers = custom ``` and my logs ``` prefect kevinkho Kevins MacBook Pro sources python query py 2022 02 24 14 51 57 0500 INFO prefect FlowRunner | Beginning Flow run for 2022 02 24 14 51 57 0500 INFO prefect TaskRunner | Task abc Starting task run 2022 02 24 14 51 57 0500 INFO custom | This is an info message 2022 02 24 14 51 57 0500 DEBUG custom | This is an debug message 2022 02 24 14 51 57 0500 INFO prefect TaskRunner | Task abc Finished task run for task with final state Success 2022 02 24 14 51 57 0500 INFO prefect FlowRunner | Flow run SUCCESS all reference tasks succeeded```

Kevin Kho · Answer

Without the config toml entry it is just ``` 2022 02 24 15 00 03 0500 INFO prefect FlowRunner | Beginning Flow run for 2022 02 24 15 00 03 0500 INFO prefect TaskRunner | Task abc Starting task run 2022 02 24 15 00 03 0500 INFO prefect TaskRunner | Task abc Finished task run for task with final state Success 2022 02 24 15 00 03 0500 INFO prefect FlowRunner | Flow run SUCCESS all reference tasks succeeded```

Milton · Answer

hmm may I ask when this feature was introduced

Kevin Kho · Answer

i joined Prefect around 0 14 4 and I feel it was already there Are you before that

Kevin Kho · Answer

it was fixed in 0 9 6

Milton · Answer

oh i found it

Milton · Answer

Milton · Answer

yeah 0 9 6

Milton · Answer

We are on 0 14 11 and using a custom agent with `run config` Could you point me to the code where it is implemented for other types of agents I wonder if we need to implement this passdown logic ourselves in our own custom agent

Kevin Kho · Answer

So you can do it in the RunConfig with ```flow run config = RunConfig env= PREFECT LOGGING EXTRA LOGGERS custom ``` and that should work for all agents at least for the ones we have off the shelf This pass down logic should happen when you merge the env from the RunConfig

Kevin Kho · Answer

You can trace <https github com PrefectHQ prefect blob master src prefect agent docker agent py L367|this> line for the RunConfig enviroment pass down for the Docker Agent for example

Milton · Answer

If I pass it in as an env to `run config` it does work but it doesn t if I put it in `config toml` as the doc suggests ``` logging Extra loggers for Prefect log configuration extra loggers = custom ``` as this is out of side `run config` and doesn t get picked up

Milton · Answer

Would you mind clarifying how you supply `config toml`

Milton · Answer

is this an agent level flow level or flow run level setting

Kevin Kho · Answer

I think you can pass the env on the agent side with `prefect agent start env PREFECT LOGGING ` The `config toml` is parsed when Prefect is imported so if you have a Flow with a container it would have to go in the container

Kevin Kho · Answer

By putting the env on the agent it gets passed to the flow it deploys

Milton · Answer

I specify the extra loggers in my user level `config toml` which takes precedence over the default in the container

Milton · Answer

but the flow doesn t seem to pick it up

Milton · Answer

the run configs do get picked up from this user level config toml though

Kevin Kho · Answer

What is your agent setup Is it a container based

Milton · Answer

yes it is

Kevin Kho · Answer

You already have the `config toml` inside the container and it doesn t get picked up

Milton · Answer

the agent itself runs in a container but all the flow runs are running in a container as well

Milton · Answer

i meant i already have the config toml in the flow runs containers

Kevin Kho · Answer

Let me read some code to be sure where it gets picked up one sec

Milton · Answer

i don t have a user level config toml override in the agent container

Milton · Answer

sure thanks Kevin

Kevin Kho · Answer

Can t immediate find if it s the agent `config toml` or container `config toml` that carries it over Will test later tonight

Milton · Answer

the reason we would like to put this in the flow run container is that different flows can use different loggers and this way gives us the maximum flexibility

Kevin Kho · Answer

I can replicate that `config toml` inside the container did not work but RunConfig env variable work I ll need to ask some people on the team what the intended behavior is

Milton · Answer

No problem please keep me posted Kevin

Milton · Answer

< Kevin Kho> another related issue it seems that if I do `import prefect config` in the flow the user configs I provide is not available either which kind of contradicts what it says at

Kevin Kho · Answer

Team members just got back to me and `config toml` only works for local agent You need to use the environment variable for the container based Flows About this let me test quickly

Milton · Answer

i guess it is probably due to the same reason

Kevin Kho · Answer

So yeah I do see it on local with ```from prefect import config print dict config ``` but don t know about Flows with backend in a container readily

Milton · Answer

if it is not mentioned could you help update the docs to make it clear It took us a day to find out in this hard way only with your help

Milton · Answer

Also i wonder if there is a reason not allowing flows to use config toml Can it be made happen

Kevin Kho · Answer

Dunno yet will open a ticket Because I think we are also moving away from `config toml` for Prefect 2 0 Will open an issue later to track

Milton · Answer

oh that s interesting to know

Milton · Answer

what will come in place of that then

Kevin Kho · Answer

It seems the feature hasn t been announced yet but will be in like 2 weeks time I think

Milton · Answer

looking forward to it