@Marvin do you have any updates on this error message for prefect 3.x hosted

10:55:27.270 | ERROR   | prefect.server.services.flowrunnotifications - Error sending notification for policy 0c740667-8ad8-4357-bf4a-2165165d625b on flow run b69196ac-3e1a-4874-8a5d-a66f5fb21b6a
Feb 06 10:55:27  prefect[118686]: 10:55:27.260 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 0 pending items.

The error message you're encountering in Prefect 3.x seems related to issues with the

GlobalEventLoopThread

and the

EventsWorker

service. Here are some insights based on the gathered information: 1. Error Context: The error appears when Prefect's internal concurrency service, specifically the

EventsWorker

, fails. This might be related to the notification system or task run registration issues. 2. Related Issues: There have been reports of similar issues in the Prefect GitHub repository. For instance, a user reported task run completions not being registered correctly, which might be related to how Prefect's concurrency and event systems are managed (GitHub Issue). 3. Documentation: Prefect's documentation on setting up alerts and automations might be helpful if this error is related to notification policies. You can explore more on setting up alerts here. 4. Potential Solutions: - Version Check: Ensure you are using a stable release of Prefect 3.x, as some release candidates had unresolved issues. - Configuration: Review your notification and concurrency configurations. Sometimes, misconfigurations can lead to such errors. If these suggestions don't resolve your issue, you might consider reaching out to Prefect's community forums or GitHub for more targeted support.

@Marvin do this help you find anything else on what may be causing this issue?

Feb 06 10:02:18 SRV- prefect[2505359]: 10:02:18.071 | ERROR   | prefect.server.services.flowrunnotifications - Error sending notification for policy 631ba30d-4339-49fe-bd1e-3c01a9d004e9 on flow run d5a0bef6-468b-49e6-9c60-6e998515d8e6
Feb 06 10:02:18 SRV- prefect[2505359]: RuntimeError: Cannot put items in a stopped service instance.
Feb 06 10:02:18 SRV- prefect[2505359]:     raise RuntimeError("Cannot put items in a stopped service instance.")
Feb 06 10:02:18 SRV- prefect[2505359]:   File "/home/prefect/production/server/.virtualenv/lib/python3.12/site-packages/prefect/_internal/concurrency/services.py", line 109, in send
Feb 06 10:02:18 SRV- prefect[2505359]:     worker_instance.send(event_obj)
Feb 06 10:02:18 SRV- prefect[2505359]:   File "/home/prefect/production/server/.virtualenv/lib/python3.12/site-packages/prefect/events/utilities.py", line 85, in emit_event
Feb 06 10:02:18 SRV- prefect[2505359]:     emit_event(event=f"{kind}.loaded", resource=resource, related=related)
Feb 06 10:02:18 SRV- prefect[2505359]:   File "/home/prefect/production/server/.virtualenv/lib/python3.12/site-packages/prefect/blocks/core.py", line 704, in _from_block_document

The error message you're experiencing, "Cannot put items in a stopped service instance," is related to threading issues within Prefect's

QueueService

subclasses, such as

EventsWorker

. Here are some insights based on the findings: 1. Known Issue: This problem has been identified in Prefect's GitHub repository, where threading issues with

QueueService

implementations were reported. Specifically, the error occurs when trying to interact with a service instance that has already been stopped (GitHub Issue). 2. Potential Cause: The error is likely related to how Prefect manages concurrency and threading. The

EventsWorker

service might be stopping prematurely or not restarting as expected, leading to this runtime error. 3. Configuration Review: You might want to review your Prefect server and service configurations. Ensure that service settings, especially related to concurrency and notifications, are correctly configured. Documentation on service settings can be found here. 4. Version and Environment: Double-check that you are using a stable version of Prefect and that your environment matches the required dependencies. Sometimes, mismatches or outdated versions can exacerbate such issues. If these insights don't resolve the issue, consider reaching out to Prefect's support channels or community forums for more tailored assistance.