Hey @Marvin , whats going wrong if I get the following Prefect Error: "GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed " ?

The error message you're encountering, "GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed", may relate to a condition in Prefect 2.x where the agent encounters issues with submitting flow runs due to client-side timeouts. From the GitHub issue found, here's a summary of what could be happening: - An agent attempts to submit a flow run, proposing a

Pending

state. - It's possible for the client to timeout, but the server operation might still succeed. - As a result, the flow run enters a

Pending

state on the server side. - However, because the client encountered a timeout error, it may raise a retry-able

httpx.ReadTimeout

error. - When the client retries the request, the server may deny the state transition because the flow run is already in a

Pending

state, and this can leave the flow run stuck. The issue on GitHub suggests that this behavior could potentially be mitigated by increasing the client-side timeout setting

PREFECT_API_REQUEST_TIMEOUT

. Here are the agent logs that you may see associated with this scenario:

plaintext
INFO | prefect.agent - Submitting flow run '<FLOW-RUN-ID>'"

...

DEBUG | prefect.client - Encountered retryable exception during request. Another attempt will be made in <FLOAT>s. This is attempt <ATTEMPT>/6.

Traceback (most recent call last):
...
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2576)
...
The above exception was the direct cause of the following exception:
...
httpx.ReadTimeout

...

INFO | prefect.agent - Aborted submission of flow run '<FLOW-RUN-ID>'. Server sent an abort signal: This run is in a PENDING state and cannot transition to a PENDING state.

In this context, if you're facing this error, consider: - Checking if your agent is timing out when submitting flow runs. - Looking at the agent and server logs to better understand the timing and error. - Possibly increasing the

PREFECT_API_REQUEST_TIMEOUT

configuration. If this doesn't solve your issue, you might want to provide more detailed logs or error messages, seek additional support from the Prefect community or support channels, or check for updates or patches from Prefect that might address this issue.