I'm using Prefect v3 to orchestrate some ETL pipelines. Works great and reliably, but I have not been able to solve logging. I have many loggers in my codebase that emit important information that I need to be captured and displayed in the UI. The only way I've learned to accomplish this would be to manually add the name of each individual logger to the extra loggers config. But this feels wrong to me. Surely there is a way to capture all the same logging and print statements (ie, all the output) that I see in a terminal run in the prefect UI. If someone could help me out with this it would be much appreciated. I've spent more time than I'd like trying to solve this already and gotten nowhere; feel like I'm missing something. Thanks!

hi @Adam Lineberry I've been meaning to make a better doc on this but does this help? https://prefect-community.slack.com/archives/CL09KU1K7/p1729623248708589?thread_ts=1729616324.217149&cid=CL09KU1K7 i.e. customize your

logging.yaml

file and then set the setting to tell prefect to use it

Thanks @Nate. I worked on this a bit today and didn't quite solve it. Is the idea to use logfire to capture the standard python logs and then capture the logfire logs with prefect using the logging.yml? I didn't have logfire installed, so I did that. I created the file `/home/adam/.prefect/logging.yml`:

handlers:

  # Add Logfire handler
  logfire:
    level: 0
    class: logfire.LogfireLoggingHandler
    formatter: standard

loggers:
  prefect:
    level: "${PREFECT_LOGGING_LEVEL}"

  prefect.flow_runs:
    level: NOTSET
    handlers: [api, logfire] # Added logfire

  prefect.task_runs:
    level: NOTSET
    handlers: [api, logfire] # Added logfire

I set the environment variables:

PREFECT_LOGGING_LEVEL='INFO'
PREFECT_LOGGING_CONFIG_PATH='/home/adam/.prefect/logging.yml'

Added the logfire configure line in the .py file that defines the tasks and flow:

from prefect import flow, task
import logfire


logfire.configure(send_to_logfire=False, config_dir=os.environ['...'])


@task
def ingest_task():
    ...


@task
def train_model_task():
   ...


@flow
def nightly_flow():
    ingest_task()
    train_model_task()

And then serve that flow with another .py file

from prefect import serve

from nightly.pilot_model_nightly import nightly_flow


if __name__ == '__main__':
    serve(
        nightly_flow.to_deployment(name='pilot_model_nightly', cron='0 9 * * 1-5')
    )

so just to be clear I wasnt suggesting that you need to use logfire at all, I was just showing how you can define your own logging.yaml file and then reference it as a setting if you have some set of external libraries that define python loggers and want to include them, then this is the best way that Im familiar with > The only way I've learned to accomplish this would be to manually add the name of each individual logger to the extra loggers config

@Nate I've looked into this some more and I understand what I misunderstood about logfire. Ok, putting logfire aside. Help me understand how this is supposed to work. I have all these different loggers using a

logging.StreamHandler

. The only commonality they have is that they all propagate up to the root logger. It seems to me like I need to somehow hook into the root logger and forward all log records it receives to the prefect logger in order to accomplish what I'm trying to. How does the solution you proposed accomplish this? I don't understand how adding a new handler to the prefect loggers addresses logging messages from my other loggers and these loggers are disconnected and adding a handler doesn't connect them. Thanks for your help.