Hi everyone is there a way I can use FileHandler or other me

Question

Hi everyone is there a way I can use FileHandler or other means to write log to a log file when I use Prefect Cloud + Local Agent The FileHandler works when I run locally with flow run but it writes nothing if I register the flow and run it on Prefect Cloud

Tuoyi Zhao · Answer

below are test code i use

Tuoyi Zhao · Answer

import prefect from prefect import task Flow Parameter import logging task log stdout=True def log my stdout print I will be logged with Flow testlog as flow log my stdout logger = prefect utilities logging get logger handler = logging FileHandler filename=r K XX PrefectETL testlog test log handler setFormatter logging Formatter asctime s levelname s name s | message s logger addHandler handler flow run for handler in logger handlers 2 handler close logger removeHandler handler flow register project name= testlog

Kevin Kho · Answer

I think this will work if you use Script based storage because the content of the Flow block is executed during registration It builds the DAG So in your case the handler is being attached during build registration time but not during runtime

Kevin Kho · Answer

To get it to attach during runtime you can attach the handler in a Task or store your Flow as a Script S3 Github Local Script

Kevin Kho · Answer

You can find more info <https docs prefect io core idioms script based html|here> about that

Tuoyi Zhao · Answer

Thanks < Kevin Kho> I tried warpping the addHandler in a Task and it works However since the handler is executed after the flow begins so the log written to the file misses the part like 2022 02 22 15 31 49 0800 INFO prefect CloudFlowRunner | Beginning Flow run for testlog is there a workaround here besides is it recommended necessary to remove the added handler at the end of the flow

Kevin Kho · Answer

Yeah you can store as a script so the file is executed during runtime and the handler is attached at the end of the task You shouldn t need to remove the handler I think