Hi, all of our flow runs have been failing for the last 12 hours with the error

Submission failed. RuntimeError: Cannot put items in a stopped service instance.

Does anyone know what might be causing this? cc @Marvin

It seems there's no direct information available from Prefect's documentation or GitHub issues related to the error "Cannot put items in a stopped service instance". However, this error generally hints at issues with infrastructure services that have stopped unexpectedly. Here are a few steps you can take to resolve the issue: 1. Check Your Infrastructure Services: Ensure that all your infrastructure services (e.g., ECS, Kubernetes, Docker) are up and running. Restart any services that may have stopped. 2. Investigate Logs: Check the logs for your service instances and Prefect infrastructure to look for any abnormal events or messages around the time the issues started occurring. 3. Restart Prefect: Sometimes, restarting Prefect services and any associated infrastructure can resolve these types of issues. 4. Prefect Community and Support: If the issue persists, consider reaching out to the Prefect community on Slack or raising a support ticket. You can find Prefect ECS integration documentation here: https://docs.prefect.io/latest/integrations/prefect-aws/ecs_worker/ If you need more assistance, feel free to ask!

I also see this error in the prefect agent logs:

ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 4 pending items.

Do you know anything useful about that @Marvin?

It looks like the error "GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with N pending items" has come up in Prefect GitHub issues. This issue occurs intermittently and doesn't seem to impact functionality directly, but it does indicate that the EventsWorker service within Prefect is failing to process some events, which might be causing your flow runs to fail. One relevant GitHub issue is here: Service 'EventsWorker' failed with N pending items ### Suggested Steps: 1. Check Resource Availability: Ensure your compute resources and network are stable. 2. Restart Prefect Workers: Sometimes restarting the workers can clear up the pending items and reset the service. 3. Monitor Prefect Logs: Keep an eye on the logs for any additional errors or patterns that might give more clues. If the issue persists, consider reaching out via the Prefect Slack community or opening a new GitHub issue for more personalized support. For more detailed guidance on work pools and workers, refer to Prefect’s documentation here: https://docs.prefect.io/latest/concepts/work-pools/ If you need any further assistance, just let me know!