Hey all, we are seeing the following exception quite a bit on prefect 3, self-hosted server. What could be the reason and how to change settings or avoid it?

│ prefect.server.events.ordering.MaxDepthExceeded: occurred=DateTime(2025, 2, 16, 8, 53, 24, 212117, tzinfo=Timezone('UTC')) event='prefect.flow-run.Cancelling' resource=Resource(root={'<http://prefect.st|prefect.st> │

@Marvin, any thoughts?

The exception

prefect.server.events.ordering.MaxDepthExceeded

you're encountering on your self-hosted Prefect 3 server might be related to the limitations on the depth of event processing in your system. This could occur due to a complex or nested event structure, possibly when dealing with flow cancellations or similar operations. Here are some insights and potential solutions: 1. Event Depth Limitation: This error might suggest that the event system is encountering a recursive event structure that exceeds the maximum allowed depth. You can adjust the settings related to event processing in your Prefect server configuration. Specifically, look for settings that control the depth or complexity of event processing. 2. Configuration Adjustments: You can modify the following environment variables to adjust the event processing limits: - `PREFECT_SERVER_EVENTS_MAXIMUM_RELATED_RESOURCES`: Controls the maximum number of related resources an event can have. - `PREFECT_SERVER_EVENTS_MAXIMUM_SIZE_BYTES`: Controls the maximum size of an event when serialized to JSON. You can find more about these settings in the Prefect 3.x documentation. 3. Known Issues: There is a known issue related to race conditions causing sub-deployments to be stuck in a "Canceling" state. This might be relevant if your flows involve sub-deployments and cancellations. You can explore this issue further on GitHub. By adjusting the configuration settings or reviewing the structure of your flows and events, you may mitigate the occurrence of this exception. If the problem persists, consider reaching out to the Prefect community or support for more detailed troubleshooting.