Kyle McChesney
11/04/2025, 5:05 PMget_run_logger show up in the prefect UI. Is there a way to configure python std library loggers used within helper code to also route to prefect?Marvin
11/04/2025, 5:05 PMMarvin
11/04/2025, 5:11 PMPREFECT_LOGGING_EXTRA_LOGGERS environment variable to a comma‑separated list of logger names you want captured. For example:
- myapp to capture logs from logging.getLogger(__name__) in your helper package
- urllib3,botocore,boto3 to capture AWS SDK noise
- In ECS/Fargate, set this env var in your task definition, your ECS work pool’s environment, or your deployment’s environment so it’s present in the running task.
Example:
- Helper code
# myapp/helpers.py
import logging
logger = logging.getLogger(__name__)
def do_something():
<http://logger.info|logger.info>("hello from helper")
- Flow code
from prefect import flow
@flow
def my_flow():
from myapp.helpers import do_something
do_something()
if __name__ == "__main__":
my_flow()
- Environment variables (in your ECS task/work pool/deployment)
PREFECT_LOGGING_EXTRA_LOGGERS=myapp,urllib3,botocore
PREFECT_LOGGING_LEVEL=INFO
Notes and gotchas:
- Logs are only associated with a run if they’re emitted while a flow/task is active. If a library logs at import time before the run starts, those may not appear in the UI.
- Some libraries set propagate = False or attach their own handlers. If you still don’t see logs after setting PREFECT_LOGGING_EXTRA_LOGGERS, you can force‑attach Prefect’s handler:
import logging
from prefect.logging.handlers import APILogHandler
logger = logging.getLogger("myapp")
logger.addHandler(APILogHandler())
logger.setLevel("INFO")
logger.propagate = True
- To capture print() output in the UI as logs, set:
PREFECT_LOGGING_LOG_PRINTS=true
- For advanced control, you can provide a full logging config file and set PREFECT_LOGGING_SETTINGS_PATH to it.
Docs:
- Advanced logging customization
- How to add logging to flows and tasks
- APILogHandler reference
- Settings reference (see PREFECT_LOGGING_EXTRA_LOGGERS, PREFECT_LOGGING_LOG_PRINTS)
If you share the logger names or libraries you’re using, I can suggest a concrete PREFECT_LOGGING_EXTRA_LOGGERS value for your ECS tasks.