Craig Rose
07/31/2024, 7:49 AM{
"env": {
"PREFECT_LOGGING_EXTRA_LOGGERS": "satellite_ghrsst_l3s_1day_daynighttime_single_sensor_australia,radar_TurquoiseCoast_velocity_hourly_average_delayed_qc"
}
}
Marvin
07/31/2024, 7:50 AMPREFECT_LOGGING_EXTRA_LOGGERS
environment variable is set in the correct scope where the Prefect worker or the flow run operates. Sometimes, the environment variable might be set in a different scope, causing it to be ignored.
2. Logger Configuration:
Make sure that the loggers satellite_ghrsst_l3s_1day_daynighttime_single_sensor_australia
and radar_TurquoiseCoast_velocity_hourly_average_delayed_qc
are correctly named and initialized in your code. The names should exactly match the loggers being used in your application.
3. Compatibility Issues:
There might be issues or bugs related to specific versions. You might want to check these reported issues for more insights:
- Logs from Dask tasks or agents might not be captured properly (Issue #10829).
- Prefect logger might not work correctly with specific runners like Ray (Issue #13011).
### Steps to Troubleshoot
1. Verify Logger Names:
Ensure that the logger names specified in PREFECT_LOGGING_EXTRA_LOGGERS
match the loggers used in your application code.
2. Check Environment Variables:
Verify that the environment variables are indeed set correctly in the environment where Prefect processes are running. You can add logging or print statements to check the environment variables in the runtime.
3. Update Prefect Version:
Ensure that you are using the latest version of Prefect as some of these issues might have been resolved in newer versions.
### Example
Here's how to log from additional loggers in your Prefect Flow:
python
import logging
from prefect import flow, get_run_logger
# Configure extra loggers normally
extra_logger1 = logging.getLogger('satellite_ghrsst_l3s_1day_daynighttime_single_sensor_australia')
extra_logger2 = logging.getLogger('radar_TurquoiseCoast_velocity_hourly_average_delayed_qc')
# Log from these loggers
<http://extra_logger1.info|extra_logger1.info>("This is a test log from satellite logger.")
<http://extra_logger2.info|extra_logger2.info>("This is a test log from radar logger.")
@flow
def my_flow():
logger = get_run_logger()
<http://logger.info|logger.info>("This is a test log from the flow logger.")
<http://extra_logger1.info|extra_logger1.info>("This is another test log from satellite logger within the flow.")
<http://extra_logger2.info|extra_logger2.info>("This is another test log from radar logger within the flow.")
if __name__ == "__main__":
my_flow()
### References
- Guides on logging Prefect Logging Guide
- Overriding Job Variables
- GitHub Issue #10829
- GitHub Issue #13011
You might want to verify these areas to ensure the logging works as expected.