https://prefect.io logo
f

Felix Schran

01/25/2021, 1:59 PM
Hi all, in prefect core how can I save all the aumatic logging messages to a file? First (before I run the flow), I create a logging file by calling loguru.logger via logger.add("logs.txt"). logger.info("Some info message before the flow is run") then writes "Some info message before the flow is run" to the file build.txt. Next, I run the flow and then get all these nice logging messages about the tasks. For instance: [2021-01-25 12:30] INFO - prefect.TaskRunner | Task 'xyz': Finished task run for task with final state: 'Success'... How can I add these logging messages to the logging file "logs.txt" from above? So ideally my build.txt file should include in the end: ... | Some info message before the flow is run. ... | Task 'xyz': Finished task run for task with final state: 'Success'...
d

Dylan

01/25/2021, 3:59 PM
Hi @Felix Schran! I’d like to learn a little more about your objective. First, are these runs orchestrated with Prefect Cloud/Server? Or are these run using only Prefect Core? You can add all the logs you want! For more on Prefect Loggers, check out: https://docs.prefect.io/core/concepts/logging.html https://docs.prefect.io/core/idioms/logging.html
z

Zanie

01/25/2021, 4:00 PM
I’ve answered this before and have a basic example — the key here is
addHandler
.
Copy code
import os
from logging import FileHandler

import prefect
from prefect.environments.storage import Local
from prefect import task, Flow


LOG_PATH = os.path.expanduser("~/flow-logs.txt")


def get_logger():
    logger = prefect.context.get("logger")
    logger.addHandler(FileHandler(LOG_PATH))
    return logger


@task()
def report_start_day():
    logger = get_logger()
    <http://logger.info|logger.info>(prefect.context.today)
    <http://logger.info|logger.info>(prefect.context.flow_run_id)


with Flow("flow-that-logs", storage=Local()) as flow:
    report_start_day()

flow.register(project_name="default")
💯 1
f

Felix Schran

01/25/2021, 5:38 PM
I am only using prefect core without the prefect cloud backend. Thanks for the example @Zanie When I run that example, I get the output printed: [2021-01-25 173558+0000] INFO - prefect.FlowRunner | Beginning Flow run for 'flow-that-logs' [2021-01-25 173558+0000] INFO - prefect.TaskRunner | Task 'report_start_day': Starting task run... [2021-01-25 173558+0000] INFO - prefect.report_start_day | 2021-01-25 [2021-01-25 173558+0000] INFO - prefect.report_start_day | 4991f380-b29f-4f59-a1a1-295f98fd1c18 [2021-01-25 173558+0000] INFO - prefect.TaskRunner | Task 'report_start_day': Finished task run for task with final state: 'Success' [2021-01-25 173558+0000] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded I would like to exactly write that output into the log file "flow-logs.txt". How can I do that?
z

Zanie

01/26/2021, 4:42 PM
If you’re only using core without the backend then you can just add the file handler at the global scope (before calling flow.run()) rather than in a function.
Python supports full customization of handler for existing loggers. Please see https://docs.python.org/3/howto/logging.html