@Marvin If I include the following in Job Variables section of Custom run the extra logs are not appearing on the console. What could be wrong

Understood, you are trying to include extra logs in the console output by adjusting the Job Variables in Custom run via Prefect, but they're not appearing. There could be several reasons for this issue, so let’s approach it step-by-step. 1. Configuration Issue: - The environment variable for logging might not be set correctly. - Ensure your logging configuration in your Prefect

Job

is set up correctly. 2. Logging Setup in Code: - There could be an incorrect logging configuration in your Python code itself. - Make sure you're configuring the logging module to output to the console. 3. Permissions & Environment: - Your job might not have the right permissions or environment setup to log to the console. Let's dive into the documentation to understand the best practices for setting logging in Prefect and what might be missing or incorrectly configured. I'll search for relevant information related to setting up and troubleshooting logging in Prefect. To help you figure out why your extra logs are not appearing on the console, here are some insights and potential solutions: 1. Logging Configuration: Prefect uses Python’s standard logging library. Ensuring your logging configuration is adequately set up can solve your issue. You can customize logging by creating your custom logging configuration file (

logging.yml

) and setting the

PREFECT_LOGGING_SETTINGS_PATH

to point to this file. More on logging configuration: https://docs.prefect.io/latest/guides/logs/ 2. Custom Run and Job Variables: You can pass custom variables in your flow run infrastructure by using job variables. Ensure these variables are properly defined and utilized in your job configuration. More on overriding job variables: https://docs.prefect.io/latest/guides/deployment/overriding-job-variables/ 3. Logging in Prefect: Here’s an example of how to set up logging in your Prefect flow:

python
   from prefect import flow, get_run_logger

   @flow(name="log-example-flow")
   def logger_flow():
       logger = get_run_logger()
       <http://logger.info|logger.info>("INFO level log message.")

This way you can send messages to the logger in both flows and tasks. Check a complete example on logging: https://raw.githubusercontent.com/zzstoatzz/prefect-code-examples/main/examples/tasks/basic.py By ensuring that your logging configuration is correctly set up and the job variables are properly specified, your extra logs should appear on the console as expected. If the issue persists, feel free to provide more details, and we can troubleshoot further.