Hi everyone, is there a way I can use FileHandler or other means to write log to a log file when I use Prefect Cloud + Local Agent? The FileHandler works when I run locally with flow.run() but it writes nothing if I register the flow and run it on Prefect Cloud.
Tuoyi Zhao
02/22/2022, 10:50 PM
below are test code i use:
Tuoyi Zhao
02/22/2022, 10:50 PM
import prefect
from prefect import task, Flow, Parameter
import logging
@task(log_stdout=True)
def log_my_stdout():
print("I will be logged!")
with Flow('testlog') as flow:
log_my_stdout()
logger = prefect.utilities.logging.get_logger()
handler = logging.FileHandler(filename=r'K:\XX\_PrefectETL\testlog\test.log')
handler.setFormatter(logging.Formatter("[%(asctime)s] %(levelname)s - %(name)s | %(message)s"))
logger.addHandler(handler)
flow.run()
for handler in logger.handlers[2]
handler.close()
logger.removeHandler(handler)
# flow.register(project_name="testlog")
k
Kevin Kho
02/22/2022, 10:54 PM
I think this will work if you use Script based storage because the content of the Flow block is executed during registration. It builds the DAG. So in your case, the handler is being attached during build/registration time but not during runtime.
Kevin Kho
02/22/2022, 10:54 PM
To get it to attach during runtime, you can attach the handler in a Task or store your Flow as a Script (S3, Github, Local Script)
Thanks @Kevin Kho! I tried warpping the addHandler in a Task, and it works! However, since the handler is executed after the flow begins, so the log written to the file misses the part like '[2022-02-22 153149-0800] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'testlog''......is there a workaround here? besides, is it recommended / necessary to remove the added handler at the end of the flow?
k
Kevin Kho
02/23/2022, 12:14 AM
Yeah you can store as a script so the file is executed during runtime and the handler is attached at the end of the task. You shouldn’t need to remove the handler I think
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.