Tuoyi Zhao

    Tuoyi Zhao

    7 months ago
    Hi everyone, is there a way I can use FileHandler or other means to write log to a log file when I use Prefect Cloud + Local Agent? The FileHandler works when I run locally with flow.run() but it writes nothing if I register the flow and run it on Prefect Cloud.
    below are test code i use:
    import prefect from prefect import task, Flow, Parameter import logging @task(log_stdout=True) def log_my_stdout():     print("I will be logged!") with Flow('testlog') as flow:     log_my_stdout()     logger = prefect.utilities.logging.get_logger()     handler = logging.FileHandler(filename=r'K:\XX_PrefectETL\testlog\test.log')     handler.setFormatter(logging.Formatter("[%(asctime)s] %(levelname)s - %(name)s | %(message)s"))     logger.addHandler(handler) flow.run() for handler in logger.handlers[2:]:     handler.close()     logger.removeHandler(handler)

    flow.register(project_name="testlog")

    Kevin Kho

    Kevin Kho

    7 months ago
    I think this will work if you use Script based storage because the content of the Flow block is executed during registration. It builds the DAG. So in your case, the handler is being attached during build/registration time but not during runtime.
    To get it to attach during runtime, you can attach the handler in a Task or store your Flow as a Script (S3, Github, Local Script)
    You can find more info here about that
    Tuoyi Zhao

    Tuoyi Zhao

    7 months ago
    Thanks @Kevin Kho! I tried warpping the addHandler in a Task, and it works! However, since the handler is executed after the flow begins, so the log written to the file misses the part like '[2022-02-22 15:31:49-0800] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'testlog''......is there a workaround here? besides, is it recommended / necessary to remove the added handler at the end of the flow?
    Kevin Kho

    Kevin Kho

    7 months ago
    Yeah you can store as a script so the file is executed during runtime and the handler is attached at the end of the task. You shouldn’t need to remove the handler I think