Hi y'all. I have a script running that seems to be functioning alright — the logs in Cloud don't any issues. But my piped output shows

16:50:25.956 | ERROR   | prefect._internal.concurrency.services - Service 'EventsWorker' failed.

Any idea what's going on?

Can you share the traceback please?

It's happening multiple times, but the initial subflows have completed successfully. This is from the latest subflow that Cloud shows as still running

16:50:26.347 | INFO    | Flow run 'armored-swan' - Executing 1 SQL statements
16:50:26.422 | ERROR   | prefect._internal.concurrency.services - Service 'EventsWorker' failed.
Traceback (most recent call last):
  File "/home/.../.venv/lib/python3.10/site-packages/prefect/_internal/concurrency/services.py", line 113, in _run
    async with self._lifespan():
  File "/home/dair/.pyenv/versions/3.10.0/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/home/.../.venv/lib/python3.10/site-packages/prefect/events/worker.py", line 30, in _lifespan
    async with self._client:
  File "/home/.../.venv/lib/python3.10/site-packages/prefect/events/clients.py", line 118, in __aenter__
    await self._reconnect()
  File "/home/.../.venv/lib/python3.10/site-packages/prefect/events/clients.py", line 136, in _reconnect
    self._websocket = await self._connect.__aenter__()
  File "/home/.../.venv/lib/python3.10/site-packages/websockets/legacy/client.py", line 637, in __aenter__
    return await self
  File "/home/.../.venv/lib/python3.10/site-packages/websockets/legacy/client.py", line 655, in __await_impl_timeout__
    return await self.__await_impl__()
  File "/home/.../.venv/lib/python3.10/site-packages/websockets/legacy/client.py", line 662, in __await_impl__
    await protocol.handshake(
  File "/home/.../.venv/lib/python3.10/site-packages/websockets/legacy/client.py", line 329, in handshake
    raise InvalidStatusCode(status_code, response_headers)
websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 403

This is an error in a background service so it wont fail your runs

OK thanks. So, I shouldn't worry about it?

You’ll be missing events since the events service is failing

cc: @Chris Guidry