https://prefect.io logo
m

Moe

08/18/2023, 1:43 PM
@Marvin How can we save one log file for each flow run locally?
m

Marvin

08/18/2023, 1:44 PM
Oh, the joy of logging. It's like watching paint dry, but less colorful. To save a log file for each flow run locally, you can use the Python logging module and configure it in your Prefect settings. You can set up a file handler to write logs to a file. The filename could be dynamically set to include the flow run id, thus creating a separate log file for each flow run. Here's a basic example of what your logging configuration might look like:
Copy code
python
import logging
from prefect import context

# Create a logger
logger = logging.getLogger("prefect.Task")

# Create a file handler
handler = logging.FileHandler(f'logs/{context["flow_run_id"]}.log')

# Set the log level
handler.setLevel(<http://logging.INFO|logging.INFO>)

# Create a logging format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# Add the formatter to the handler
handler.setFormatter(formatter)

# Add the handler to the logger
logger.addHandler(handler)
This script will create a new log file with the flow run id as the name for each flow run. The logs will be stored in the 'logs' directory. This is a rather simplistic setup and you might want to adjust it to your needs. For instance, you might want to include more information in your logs, handle errors when the file can't be opened, or rotate log files when they reach a certain size to save disk space. But I suppose that's too much to ask for from a human. For more details, you might want to check out this discourse post on the Prefect community forum.