https://prefect.io logo
p

Patrick Tan

04/08/2022, 6:48 PM
Prefect logging question: When I ran below from Mac, both info and debug rows (2) were displayed, but when I registered and ran flow from Cloud, only 1 row, the logger.info row is shown, please advise logger=prefect.context.get("logger") logger.setLevel(logging.DEBUG) logger.info(‘this is test info ) logger.debug('this is test debug')
k

Kevin Kho

04/08/2022, 6:49 PM
Did you set the logging level on the Flow?
p

Patrick Tan

04/08/2022, 6:50 PM
No
k

Kevin Kho

04/08/2022, 6:52 PM
You can do it on the RunConfig
Copy code
flow.run_config = RunConfig(..., env={"PREFECT__LOGGING__LEVEL": "DEBUG"}
p

Patrick Tan

04/08/2022, 7:14 PM
Not working: I ran same flow locally and cloud and put them side by side , as you can see the 2 DEBUG messages is not showing in cloud
k

Kevin Kho

04/08/2022, 7:20 PM
This will be hard for me to guess without seeing more code. Could you give me a small example of something I can run?
p

Patrick Tan

04/08/2022, 7:41 PM
Copy code
from prefect import Flow, task
from prefect.executors import LocalExecutor
from prefect.run_configs import LocalRun
import prefect
import logging


def get_logger():
    logger=prefect.context.get('logger')
    logger.setLevel(logging.DEBUG)
    <http://logger.info|logger.info>('task INFO')
    logger.debug('task DEBUG')
    return logger

def get_flow():
    get_logger_task = task(get_logger)
    with Flow("logger-flow") as f:
        get_logger_task()
        f.run_config = LocalRun(labels=['qa'])
    return f

flow = get_flow()


if __name__ == "__main__":
    flow.executor = LocalExecutor()
    flow.register("default")
    flow.run()
compare output running from Cloud after registered, and ran locally
k

Kevin Kho

04/08/2022, 7:56 PM
Thanks! Will try and explore this in a bit
But if change your flow to add the logging level I get:
Copy code
def get_flow():
    get_logger_task = task(get_logger)
    with Flow("logger-flow") as f:
        get_logger_task()
        f.run_config = LocalRun(env={"PREFECT__LOGGING__LEVEL": "DEBUG"})
    return f
So the logger is created from context and I think any changes to it will only persist inside a task. I suggest you just use the global logging level. Are you trying to do something that requires multiple logging levels?
p

Patrick Tan

04/11/2022, 12:42 PM
I like to log detail information in debug level. In the case there is issue, I can set to debug level so I can troubleshoot.
k

Kevin Kho

04/11/2022, 1:59 PM
I don’t think the logging configuration stops that? You can just toggle the logging level to debug when you want to see debug level logs
p

Patrick Tan

04/11/2022, 5:13 PM
Got it, Thanks!
In the same topic. I have parent flow that run child flow using create_flow_run.map... how do I propagate same log level from parent flow to child flow?
Copy code
create_flow_run.map(
    flow_name=unmapped("livelots flow"),
    project_name=unmapped("LiveLots-ETL"),
    idempotency_key=idempotency_keys,
    parameters=filelist,
)
k

Kevin Kho

04/11/2022, 6:09 PM
The only way would be by adding a RunConfig with an env variable.
create_flow_run
takes in a RunConfig
p

Patrick Tan

04/11/2022, 6:18 PM
It is working! Thanks again!
k

Kevin Kho

04/11/2022, 6:37 PM
Of course!
4 Views