@Marvin Can I use python's logger via

import logger

to log messages to the prefect dashboard or do I need to use prefect's implementation of the logger tool?

You should use Prefect's logger to log messages that will appear in the Prefect dashboard. Importing a logger using

import logger

from Python's standard logging module won't automatically capture logs to Prefect's backend. Here’s how you can use Prefect's logger:

python
from prefect import flow, get_run_logger

@flow
def my_flow():
    logger = get_run_logger()
    <http://logger.info|logger.info>("This is a log message that will appear in the Prefect dashboard.")

my_flow()

Using

get_run_logger()

ensures that logs are captured and displayed in the Prefect UI. For more information, check the Prefect docs. If you have any further questions, feel free to ask!

@Marvin If I used the logger from python's standard logging module, how would I get logs to appear in the dashboard?

To get logs from Python's standard logging module to appear in the Prefect dashboard, you'll need to configure your logging to forward logs to Prefect's API. You can do this by customizing the logging configuration. Here's how you can achieve this: 1. Create a Custom Logging Configuration: Prefect uses a YAML file to configure logging. You can create a

logging.yml

file with the desired configuration. 2. Set the Logging Configuration Path: Use the

PREFECT_LOGGING_SETTINGS_PATH

environment variable to point to your custom logging configuration file. 3. Forward Logs to Prefect: Ensure your configuration forwards logs to Prefect's API. You can do this by using the

APILogHandler

provided by Prefect. Here is a basic example of what your

logging.yml

file might look like:

yaml
version: 1
disable_existing_loggers: False
formatters:
  standard:
    format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(name)s - %(message)s"
handlers:
  console:
    class: logging.StreamHandler
    formatter: standard
    level: INFO
  api:
    class: prefect.logging.handlers.APILogHandler
    level: INFO
loggers:
  prefect:
    handlers: [console, api]
    level: INFO
    propagate: True
  my_custom_logger:
    handlers: [console, api]
    level: INFO
    propagate: False
root:
  handlers: [console, api]
  level: INFO

4. Example Usage: Here’s how you can use the standard logger in your flow:

python
import logging
from prefect import flow, task

@task
def my_task():
    logger = logging.getLogger("my_custom_logger")
    <http://logger.info|logger.info>("This is a log message from a task.")

@flow
def my_flow():
    logger = logging.getLogger("my_custom_logger")
    <http://logger.info|logger.info>("This is a log message from the flow.")
    my_task()

my_flow()

5. Set the Environment Variable: Make sure to set the

PREFECT_LOGGING_SETTINGS_PATH

to point to your

logging.yml

file:

sh
export PREFECT_LOGGING_SETTINGS_PATH=/path/to/logging.yml

By following these steps, your logs from the standard Python logging module should be captured and displayed in the Prefect dashboard. For more detailed logs configuration, refer to the Prefect logging guide.