Hi everyone, is there a way I can use FileHandler ...
# ask-communityt
Tuoyi Zhao
02/22/2022, 10:48 PMHi everyone, is there a way I can use FileHandler or other means to write log to a log file when I use Prefect Cloud + Local Agent? The FileHandler works when I run locally with flow.run() but it writes nothing if I register the flow and run it on Prefect Cloud.
Tuoyi Zhao
02/22/2022, 10:50 PMbelow are test code i use:
Tuoyi Zhao
02/22/2022, 10:50 PMimport prefect
from prefect import task, Flow, Parameter
import logging
@task(log_stdout=True)
def log_my_stdout():
print("I will be logged!")
with Flow('testlog') as flow:
log_my_stdout()
logger = prefect.utilities.logging.get_logger()
handler = logging.FileHandler(filename=r'K:\XX\_PrefectETL\testlog\test.log')
handler.setFormatter(logging.Formatter("[%(asctime)s] %(levelname)s - %(name)s | %(message)s"))
logger.addHandler(handler)
flow.run()
for handler in logger.handlers[2]
handler.close()
logger.removeHandler(handler)
# flow.register(project_name="testlog")
k
Kevin Kho
02/22/2022, 10:54 PM
I think this will work if you use Script based storage because the content of the Flow block is executed during registration. It builds the DAG. So in your case, the handler is being attached during build/registration time but not during runtime.
Kevin Kho
02/22/2022, 10:54 PM
To get it to attach during runtime, you can attach the handler in a Task or store your Flow as a Script (S3, Github, Local Script)
Kevin Kho
02/22/2022, 10:54 PM
You can find more info here about that
t
Tuoyi Zhao
02/22/2022, 11:36 PMThanks @Kevin Kho! I tried warpping the addHandler in a Task, and it works! However, since the handler is executed after the flow begins, so the log written to the file misses the part like '[2022-02-22 153149-0800] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'testlog''......is there a workaround here? besides, is it recommended / necessary to remove the added handler at the end of the flow?
k
Kevin Kho
02/23/2022, 12:14 AM
Yeah you can store as a script so the file is executed during runtime and the handler is attached at the end of the task. You shouldn’t need to remove the handler I think
❤️ 1
7 Views