Hi everyone, is there a way I can use FileHandler or other means to write log to a log file when I use Prefect Cloud + Local Agent? The FileHandler works when I run locally with flow.run() but it writes nothing if I register the flow and run it on Prefect Cloud.
below are test code i use:
from prefect import task, Flow, Parameter
print("I will be logged!")with Flow('testlog') as flow: log_my_stdout() logger = prefect.utilities.logging.get_logger()
handler = logging.FileHandler(filename=r'K:\XX_PrefectETL\testlog\test.log')
handler.setFormatter(logging.Formatter("[%(asctime)s] %(levelname)s - %(name)s | %(message)s"))
for handler in logger.handlers[2:]: handler.close()
7 months ago
I think this will work if you use Script based storage because the content of the Flow block is executed during registration. It builds the DAG. So in your case, the handler is being attached during build/registration time but not during runtime.
To get it to attach during runtime, you can attach the handler in a Task or store your Flow as a Script (S3, Github, Local Script)
Thanks @Kevin Kho! I tried warpping the addHandler in a Task, and it works! However, since the handler is executed after the flow begins, so the log written to the file misses the part like '[2022-02-22 15:31:49-0800] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'testlog''......is there a workaround here? besides, is it recommended / necessary to remove the added handler at the end of the flow?
7 months ago
Yeah you can store as a script so the file is executed during runtime and the handler is attached at the end of the task. You shouldn’t need to remove the handler I think