<@ULVA73B9P> Worker 'ECSWorker 03568b81-9dfc-4f67-...
# ask-community
s
@Marvin Worker 'ECSWorker 03568b81-9dfc-4f67-bca9-9cbda7485525' submitting flow run '0e3e16a7-06e2-41d2-8db4-ec9189dfbf87' 031122 PM prefect.flow_runs.worker Failed to submit flow run '0e3e16a7-06e2-41d2-8db4-ec9189dfbf87' to infrastructure. Traceback (most recent call last): File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/workers/base.py", line 1008, in _submit_run_and_capture_errors submitted_event = self._emit_flow_run_submitted_event(configuration) File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/workers/base.py", line 1306, in _emit_flow_run_submitted_event return emit_event( File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/events/utilities.py", line 87, in emit_event worker_instance.send(event_obj) File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/_internal/concurrency/services.py", line 293, in send raise RuntimeError("Cannot put items in a stopped service instance.") RuntimeError: Cannot put items in a stopped service instance. 031129 PM prefect.flow_runs.worker Reported flow run '0e3e16a7-06e2-41d2-8db4-ec9189dfbf87' as crashed: Flow run could not be submitted to infrastructure: RuntimeError('Cannot put items in a stopped service instance.')
m
thought for 21.3 seconds
Based on the error you're encountering and the related GitHub issues, this appears to be a known issue related to the events system in Prefect 3.x. The error occurs when the worker tries to emit events but the events service has been stopped. Here are a few potential solutions: 1. Temporary Workaround: Restart your worker. This will reinitialize the events service and might resolve the immediate issue. 2. Version Check: If you're using an early release candidate of Prefect 3.0, consider upgrading to the latest stable version (3.2.x+) as there have been several fixes related to the events system. 3. Configuration Adjustment: You can try setting the following environment variable to make event failures less impactful:
Copy code
bash
export PREFECT_LOGGING_EXTRA_LOGGERS='{"prefect.events": "ERROR"}'
This will reduce the noise from event-related errors while still allowing your flows to run. The good news is that while this error looks alarming, it's primarily related to the event reporting system and not the core flow execution. Your flow runs should still execute, even if the events aren't being properly recorded. If you continue to experience issues: 1. Make sure your worker has proper network connectivity to the Prefect server 2. Check if you have any firewall rules that might be interfering with the worker's communication 3. Consider running your worker with debug logging enabled to get more detailed information:
Copy code
bash
prefect worker start -p 'your-pool' --log-level DEBUG
There's an open issue tracking improvements to make event failures less disruptive: https://github.com/PrefectHQ/prefect/issues/17630