<@ULVA73B9P> - we are running all our flows in AWS...
# ask-marvin
k
@Marvin - we are running all our flows in AWS ECS (fargate tasks). Currently only things logged directly using the logger from
get_run_logger
show up in the prefect UI. Is there a way to configure python std library loggers used within helper code to also route to prefect?
m
thought for 334.5 seconds
Yes. In Prefect 3.x you can route standard library (and third‑party) loggers to the Prefect UI. What to do: - Set the
PREFECT_LOGGING_EXTRA_LOGGERS
environment variable to a comma‑separated list of logger names you want captured. For example: -
myapp
to capture logs from
logging.getLogger(__name__)
in your helper package -
urllib3,botocore,boto3
to capture AWS SDK noise - In ECS/Fargate, set this env var in your task definition, your ECS work pool’s environment, or your deployment’s environment so it’s present in the running task. Example: - Helper code
Copy code
# myapp/helpers.py
import logging

logger = logging.getLogger(__name__)

def do_something():
    <http://logger.info|logger.info>("hello from helper")
- Flow code
Copy code
from prefect import flow

@flow
def my_flow():
    from myapp.helpers import do_something
    do_something()

if __name__ == "__main__":
    my_flow()
- Environment variables (in your ECS task/work pool/deployment)
Copy code
PREFECT_LOGGING_EXTRA_LOGGERS=myapp,urllib3,botocore
PREFECT_LOGGING_LEVEL=INFO
Notes and gotchas: - Logs are only associated with a run if they’re emitted while a flow/task is active. If a library logs at import time before the run starts, those may not appear in the UI. - Some libraries set
propagate = False
or attach their own handlers. If you still don’t see logs after setting
PREFECT_LOGGING_EXTRA_LOGGERS
, you can force‑attach Prefect’s handler:
Copy code
import logging
from prefect.logging.handlers import APILogHandler

logger = logging.getLogger("myapp")
logger.addHandler(APILogHandler())
logger.setLevel("INFO")
logger.propagate = True
- To capture
print()
output in the UI as logs, set:
Copy code
PREFECT_LOGGING_LOG_PRINTS=true
- For advanced control, you can provide a full logging config file and set
PREFECT_LOGGING_SETTINGS_PATH
to it. Docs: - Advanced logging customization - How to add logging to flows and tasks - APILogHandler reference - Settings reference (see
PREFECT_LOGGING_EXTRA_LOGGERS
,
PREFECT_LOGGING_LOG_PRINTS
) If you share the logger names or libraries you’re using, I can suggest a concrete
PREFECT_LOGGING_EXTRA_LOGGERS
value for your ECS tasks.