<@ULVA73B9P> I am testing logging setup. I would l...
# ask-marvin
r
@Marvin I am testing logging setup. I would like to capture logs from external libs in Prefect UI. My test flow is : import os os.environ["PREFECT_LOGGING_EXTRA_LOGGERS"] = "lif_tests,requests" import logging import requests from prefect import flow, task def extern_library_logging_test(): # Nastavime logovanie pre requests kniznicu logging.getLogger("requests").setLevel(logging.DEBUG) # Vytvorime logger pre nasu aplikaciu logger = logging.getLogger("lif_tests") # Logujeme informaciu o testovani externych kniznic logger.info("Testujeme logovanie z externych kniznic...") # Vykoname HTTP request pomocou requests kniznice, co by malo generovat logy response = requests.get("https://httpbin.org/get", proxies={"http": "http://proxy.app.slsp.sk:8080", "https": "http://proxy.app.slsp.sk:8080"}) logger.info(f"HTTP request bol uspesny: {response.status_code}") @task def run_extern_lib(): extern_library_logging_test() @flow def exter_lib_flow_test(): run_extern_lib() if name == "__main__": logging.basicConfig(level=logging.DEBUG) exter_lib_flow_test() But I don't see logs from lif_tests and requests, why ?
m
thought for 277.5 seconds
Short answer: two common gotchas are likely biting you - Most HTTP chatter comes from urllib3, not requests — add urllib3 to EXTRA_LOGGERS. - Prefect’s API handler defaults to INFO. If you set requests to DEBUG, those DEBUG records will be dropped unless you lower Prefect’s logging level to DEBUG. Also, if you’re running this as a deployment on a worker, setting env vars inside your code is too late — set them on the worker/pool/deployment environment. What to change - Include urllib3: -
PREFECT_LOGGING_EXTRA_LOGGERS=lif_tests,requests,urllib3
- If you want DEBUG logs from those libs: -
PREFECT_LOGGING_LEVEL=DEBUG
- Avoid calling `logging.basicConfig(...)`; let Prefect manage logging. - Ensure these env vars are set before Prefect is imported in the Python process. For deployments, put them in the work pool or deployment env, not inside your flow file. Minimal example (local script)
Copy code
import os

# Set BEFORE importing Prefect
os.environ["PREFECT_LOGGING_EXTRA_LOGGERS"] = "lif_tests,requests,urllib3"
os.environ["PREFECT_LOGGING_LEVEL"] = "DEBUG"  # if you want DEBUG from urllib3/requests

import logging
import requests
from prefect import flow, task

def extern_library_logging_test():
    # Optional: make sure urllib3 emits debug
    logging.getLogger("urllib3").setLevel(logging.DEBUG)

    logger = logging.getLogger("lif_tests")
    <http://logger.info|logger.info>("Testujeme logovanie z externych kniznic...")

    response = requests.get("<https://httpbin.org/get>",
                            proxies={"http": "<http://proxy.app.slsp.sk:8080>",
                                     "https": "<http://proxy.app.slsp.sk:8080>"})
    <http://logger.info|logger.info>(f"HTTP request bol uspesny: {response.status_code}")

@task
def run_extern_lib():
    extern_library_logging_test()

@flow
def exter_lib_flow_test():
    run_extern_lib()

if __name__ == "__main__":
    # Do not call logging.basicConfig here
    exter_lib_flow_test()
If this is a deployment/worker run - Put these in the job runtime environment (examples): - In your work pool’s environment variables - In deployment config env:
Copy code
deployments:
      - name: exter-lib-test
        entrypoint: my_flow.py:exter_lib_flow_test
        work_pool:
          name: my-pool
        env:
          PREFECT_LOGGING_EXTRA_LOGGERS: "lif_tests,requests,urllib3"
          PREFECT_LOGGING_LEVEL: "DEBUG"
- Or bake them into the Docker/K8s environment Why you didn’t see the logs - requests: most logs you expect are actually emitted by urllib3 (e.g., urllib3.connectionpool). Without adding urllib3 you’ll see very little. - DEBUG vs INFO: Prefect’s API handler defaults to INFO, so DEBUG logs from requests/urllib3 are filtered unless you set
PREFECT_LOGGING_LEVEL=DEBUG
. - If this was a deployment: environment variables set inside your script come too late; Prefect configures logging on import in the worker process. Docs and references - Logging customization (how EXTRA_LOGGERS works): Logging customization - Settings reference (PREFECT_LOGGING_EXTRA_LOGGERS, PREFECT_LOGGING_LEVEL): Settings reference If you can share whether you’re running locally or via a worker/deployment, I can tailor the exact placement of these settings.
r
@Marvin Now I can see DEBUG logs from urllib3, but this log message: logger = logging.getLogger("lif_tests") logger.info("Testujeme logovanie z externych kniznic...") is not propagated to Prefect UI. Why ?
m
thought for 64.3 seconds
Because Prefect does not change the level of “extra” loggers. By default, a newly created Python logger has level WARNING, so your
<http://logger.info|logger.info>(...)
on
lif_tests
is filtered out before it ever reaches Prefect’s handler. Why urllib3 shows up but lif_tests doesn’t - You explicitly set
urllib3
(or
requests
) to DEBUG, so its records pass its own level check. - You didn’t set a level for
lif_tests
, so it stays at WARNING and drops INFO messages. Fix - Set the level on your custom logger (and ensure it’s listed in EXTRA_LOGGERS):
Copy code
# Ensure this env var is set before importing Prefect
# PREFECT_LOGGING_EXTRA_LOGGERS=lif_tests,requests,urllib3

import logging

lif_logger = logging.getLogger("lif_tests")
lif_logger.setLevel(<http://logging.INFO|logging.INFO>)  # or DEBUG
<http://lif_logger.info|lif_logger.info>("Testujeme logovanie z externych kniznic...")
Notes - Changing
PREFECT_LOGGING_LEVEL
won’t change external logger levels; it only controls Prefect’s own loggers and the API handler’s threshold. - Prefect only adds its handler to the named extra loggers; it does not modify their level or propagation. If you want, you can quickly verify the current level:
Copy code
import logging
print(logging.getLogger("lif_tests").getEffectiveLevel())  # 30 = WARNING, 20 = INFO, 10 = DEBUG
Docs: Logging customization (see “Include logs from other libraries”).
j
Also, the environment variable needs to be set before the process starts, not at runtime.