@Marvin What is best practices for printing to log? As in print number of rows being processed to log.

In Prefect, it's recommended to use the built-in logging features instead of simple print statements for a more controlled and configurable way to handle logging output. To print the number of rows being processed to the log, you would first need to import and use Prefect's logging. Here's a basic explanation with a link to the specific part of the docs that cover logging: 1. Import the logger from Prefect:

python
   from prefect import get_run_logger

2. Use the logger within your tasks and flows:

python
   @task
   def process_data(data):
       logger = get_run_logger()
       # ... process data
       num_rows_processed = len(data)
       <http://logger.info|logger.info>(f"Processed {num_rows_processed} rows.")

3. Customize the logging levels if needed. For example, to set the log level to

DEBUG

, you could use the environment variable

PREFECT_LOGGING_LEVEL="DEBUG"

. In the context of Prefect, using

<http://logger.info|logger.info>()

or the appropriate logging level function (e.g.,

logger.debug()

,

logger.warning()

, etc.) is considered a best practice for recording operational messages such as the number of rows processed. For further information on logging in Prefect and best practices, you can refer to the Prefect logging documentation. And remember, setting

log_prints=True

at the flow level will enable the capturing of print statements in the logs but using Prefect's logging system is preferred for comprehensive logging.