Aleksandr Liadov

    Aleksandr Liadov

    9 months ago
    Hi guys, My problem is following: I’d like to pass the logs (standard python logger) from the submodules to prefect logger ( so it can be displayed on the server). I understood, that it can be done with
    run config
    in flow (I do the local run with the local agent):
    run_config=UniversalRun(env={
        "PREFECT__LOGGING__EXTRA_LOGGERS": '["lib1", "lib2"]',
        "PREFECT__LOGGING__LEVEL": "DEBUG",
    })
    I can ONLY observe
    info
    level from my libs even if I try to switch on
    debug
    level! (However the prefect logger shows the information related to
    CloudFlowRunner
    from
    debug
    level without any problem). What do I wrong?
    Tom Shaffner

    Tom Shaffner

    9 months ago
    I'm having a similar problem trying to overwrite the log format
    _env_={'PREFECT__LOGGING__FORMAT':'[%(asctime)s-%(levelname)s - %(name)s]-[%(filename)s:%(funcName)s]-Line %(lineno)d: %(message)s'}
    , so mark me as also interested in this question. I've also tried it in the config.toml file to no avail.
    Kevin Kho

    Kevin Kho

    9 months ago
    This looks right to me. INFO level logs print fine for
    lib1
    ? I’ll take a look at the source, but I can’t think of any reason off the top of my head. Will try that too @Tom Shaffner
    Aleksandr Liadov

    Aleksandr Liadov

    9 months ago
    INFO
    level prints right for both libs without any problem 👍
    Tom Shaffner

    Tom Shaffner

    9 months ago
    Info is the default though, right? So if your setting is failing to overwrite you'd only see the issue when you picked something else.
    Kevin Kho

    Kevin Kho

    9 months ago
    I think he used DEBUG and got DEBUG Prefect logs but INFO logs for the
    lib1
    and
    lib2
    Aleksandr Liadov

    Aleksandr Liadov

    9 months ago
    Yes, you are right @Kevin Kho that is my problem!
    Kevin Kho

    Kevin Kho

    9 months ago
    So the source looks right to me. It just attaches the level from the config. I will make a test
    Aleksandr Liadov

    Aleksandr Liadov

    9 months ago
    Maybe it changes something, but my flow (indeed is a subflow), for which I specify
    run_config
    is run by the parent flow.
    Kevin Kho

    Kevin Kho

    9 months ago
    Ok so I tried this out:fake_lib.py
    import logging
    
    
    def function_that_logs():
        logger = logging.getLogger("foo.bar")
        <http://logger.info|logger.info>("INFO from fake_lib!")
        logger.debug("DEBUG from fake_lib!")
    theflow.py
    from fake_lib import function_that_logs
    from prefect import Flow, task
    from prefect.run_configs import LocalRun
    
    
    @task
    def foo():
        function_that_logs()
    
    
    with Flow("extra-loggers-example") as flow:
        foo()
    
    flow.run_config = LocalRun(env={"PREFECT__LOGGING__EXTRA_LOGGERS":'["foo"]',
                                    "PREFECT__LOGGING__LEVEL": "DEBUG"})
    
    flow.register("bristech")
    I guess you might need it in the RunConfig of the subflow
    Aleksandr Liadov

    Aleksandr Liadov

    9 months ago
    I should specify the same run_config for parent and subflow?
    Kevin Kho

    Kevin Kho

    9 months ago
    The subflow is a new process and subflow logs don’t make it to the parent flow so I would say you need the EXTRA_LOGGERS in any flow that uses that library
    @Tom Shaffner, so that worked when I did
    flow.run()
    .
    [2021-12-01 12:29:00-0500-INFO - prefect.FlowRunner]-[flow_runner.py:run]-Line 245: Beginning Flow run for 'extra-loggers-example'
    [2021-12-01 12:29:00-0500-INFO - prefect.TaskRunner]-[task_runner.py:run]-Line 241: Task 'foo': Starting task run...
    [2021-12-01 12:29:00-0500-INFO - prefect.TaskRunner]-[task_runner.py:run]-Line 335: Task 'foo': Finished task run for task with final state: 'Success'
    [2021-12-01 12:29:00-0500-INFO - prefect.FlowRunner]-[flow_runner.py:determine_final_state]-Line 710: Flow run SUCCESS: all reference tasks succeeded
    But I am wondering if this will translate well to Cloud because Cloud does some rendering in the UI. Are you expecting the logs tab in the UI to have this format?
    Aleksandr Liadov

    Aleksandr Liadov

    9 months ago
    Hm, It seems to me that the logs which are outside the run() method cannot be passed. In my case, we have a class which inherits the Task, so we overwrite run() to be compatible. But in the class there are also several functions in which I use logs. Those functions is called in run method and the logs from those functions I want to be able to observe. Do I smth wrong?
    Kevin Kho

    Kevin Kho

    9 months ago
    I think this should work. I can modify the example a bit. Could you give me a small example so I get a better picture?
    Tom Shaffner

    Tom Shaffner

    9 months ago
    @Kevin Kho, to your question above, yes, I was hoping to be able to see the filenames in particular in the UI; I take it the formatting it is doing prevents that? In which case maybe I need to check the logs themselves when I need this?
    Kevin Kho

    Kevin Kho

    9 months ago
    I need to ask around for this. Will get back to you when I get a response
    You can’t change how logs are rendered in the UI
    Aleksandr Liadov

    Aleksandr Liadov

    9 months ago
    @Kevin Kho It works now, I restarted my PC, and now debug level works 🤷‍♂️ Thanks for your help!