@Marvin Ive add debug level logs to my k8s job, and I see this log around every 10s

Checking for cancelled flow runs

. Can you explain more about it and if its something I can configure to a larger intervals?

The log message "Checking for cancelled flow runs" appears to be part of how Prefect checks periodically for any flow runs that have been cancelled, ensuring that the execution state is kept current and appropriate actions are taken if a cancellation is detected. The default interval for this check isn't directly documented or configurable in a user-facing way. However, it's evident from the context that this interval might be around every 10 seconds in your case. As for adjusting this interval, you would likely need to look into the underlying configurations or source code where this behavior is implemented. From my search, there doesn't appear to be a straightforward configuration option exposed to end-users for adjusting this interval. For more detailed information and possible workarounds, you might want to explore further on Prefect's GitHub issues or seek guidance directly from the Prefect community or support. Here are some related GitHub issues that might provide additional context or potential leads: 1. Prevent running late runs after some threshold: https://github.com/PrefectHQ/prefect/issues/9054 2. Unable to handle 'Cancelling' events: https://github.com/PrefectHQ/prefect/issues/12688 You may need to refer to the source code or consult with the Prefect community to confirm if there's a low-level setting that can be tweaked for your deployment. If you have further questions or need additional help, feel free to ask!

Another question @Marvin, for some reason I dont see logs any idea why?

There seem to be several common issues reported with Prefect regarding logs not showing up. Here are a few potential solutions and reasons why logs might not be visible: 1. Logging Configuration: - Ensure that you have properly configured the logging settings. You can customize the logging configuration by creating a custom

logging.yml

file. Prefect looks for this configuration file in

/.prefect/logging.yml

or a path specified by

PREFECT_LOGGING_SETTINGS_PATH

. - Refer to the Prefect logging documentation here for examples and details. 2. Prefect Loggers: - Use

get_run_logger

to access the Prefect logger within your flows and tasks. Ensure that you're using it correctly within your code. - Here’s a basic example:

python
     from prefect import flow, get_run_logger
     
     @flow(name="log-example-flow")
     def logger_flow():
         logger = get_run_logger()
         <http://logger.info|logger.info>("INFO level log message.")

- This ensures logs are captured and can be viewed in the Prefect UI. 3. Environment Variables: - Setting environment variables like

PREFECT_LOGGING_EXTRA_LOGGERS

can be crucial, especially when using additional loggers like Dask. - Example:

PREFECT_LOGGING_EXTRA_LOGGERS=dask

4. Known Issues: - There are some known issues related to logs not being visible, especially when using Dask or running flows in certain environments like Kubernetes. - Example issue: Custom logs from tasks running on a Dask cluster are not showing in the Prefect UI - For a more specific case, if you're using Dask, ensure that your setup correctly routes the logs. Here’s a GitHub issue that dives deeper into it: Dask task logs not showing in Prefect Cloud 5. Version and Updates: - Ensure that you are using a compatible and updated version of Prefect. - Verify settings related to the environment and deployment configurations, especially in containerized environments. If these steps do not resolve the issue, you may consider sharing logs, configurations, and specific environmental details with the Prefect community or support for more focused troubleshooting. For more detailed information, please refer to the official Prefect documentation and the provided GitHub issues.