@Marvin Why do I see a lot of "Flow run deleted" events in the event log? These are occurring every two minutes.

The frequent "Flow run deleted" events you're seeing every two minutes could be related to a couple of known issues in Prefect 3.x. One possibility is a race condition that affects sub-deployments, where child deployments remain in a canceling state if a parent deployment is canceled. This can cause repeated cancellations and deletions of flow runs. Another potential cause is related to deployments being launched with a specific interval that might not be completing successfully and thus getting deleted automatically. Here are some related discussions and issues: - Race condition causing sub-deployments to be stuck in Canceling state - Deployment of async function in Pending/Late stage after being launched via serve method For more detailed troubleshooting, you might want to check your deployment configurations and ensure that your flow scheduling and cancellation logic are working as intended. You can also refer to the Prefect events documentation for more insights into how events and automations are handled in Prefect. If the problem persists, consider reaching out to the Prefect community for further assistance.

@Marvin For each deletion. There is unique resource if referencing a flow run.

If each deletion event is associated with a unique resource referencing a flow run, it suggests that these events are possibly being triggered by a systematic process or automation that targets individual flow runs. Here are a few things you might consider checking: 1. Automations and Triggers: Verify if there are any automations or triggers set up in your Prefect environment that might be deleting flow runs based on certain conditions. You can refer to the Prefect automation triggers documentation for more details on how these work. 2. Deployment Configurations: Review your deployment configurations, especially any settings related to timeouts, retries, or conditions under which a flow run might be automatically canceled or deleted. 3. Event Logs: Examine the event logs closely for any patterns or additional metadata that might indicate why each flow run is being deleted. This could include specific error messages, states, or conditions prior to deletion. 4. Resource Limits: Check if there are any resource limits or quotas in place that might be causing automatic deletions when exceeded. This could be related to the number of concurrent runs, memory usage, etc. 5. Version-Specific Issues: Ensure that you are using the latest version of Prefect, as some issues might have been resolved in newer releases. You can check the latest release notes for any relevant bug fixes. If none of these avenues provide clarity, it might be helpful to engage with the Prefect community or support channels for more targeted assistance.