https://prefect.io logo
Channels
Powered by
t

Tanishq Hooda

08/14/2023, 8:31 AM
Hi All, Some of our prefect flows get stuck in Pending state and never run after that (happens intermittently). They run on EKS pods. I see this error a lot in prefect agent logs (pasted in thread), is there any bug with prefect? Using version 2.10.13
plus one 2
Copy code
08:00:07.100 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed to process item (Event(occurred=DateTime(2023, 8, 14, 8, 0, 7, 99125, tzinfo=Timezone('Etc/UTC')), event='prefect.block.kubernetes-job.get_batch_client.called', resource=Resource(__root__={'prefect.resource.id': 'prefect.block-document.my-id', 'prefect.resource.name': 'kubernetes-block'}), related=[RelatedResource(__root__={'prefect.resource.id': 'prefect.block-type.kubernetes-job', 'prefect.resource.role': 'block-type'})], payload={}, id=UUID('my-id'), follows=None), <Context object at 0x7f11cdd71e40>)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/_internal/concurrency/services.py", line 152, in _main_loop
    await self._handle(item)
  File "/usr/local/lib/python3.9/site-packages/prefect/events/worker.py", line 41, in _handle
    await self._client.emit(event)
  File "/usr/local/lib/python3.9/site-packages/prefect/events/clients.py", line 22, in emit
    return await self._emit(event)
  File "/usr/local/lib/python3.9/site-packages/prefect/events/clients.py", line 172, in _emit
    assert self._websocket
AssertionError
m

Moe

08/22/2023, 8:38 PM
@Tanishq Hooda Any updates on this? I am starting to see this issue too.
t

Tanishq Hooda

08/23/2023, 8:49 AM
nope, I have an email thread with Prefect support (@Emil Christensen is it ?)
@Moe what part of the issue do you see? Pending flows or assertionerror in log or both?
m

Moe

08/23/2023, 1:02 PM
@Tanishq Hooda My flow and subflows run and complete fine with no error in the logs. However, I am seeing these error messages pop up my local terminal where I'm running from. And the flow just continues executing despite this message and complete fine.
e

Emil Christensen

08/23/2023, 3:18 PM
@Tanishq Hooda 👋 apologies, our case system has been acting up a bit, so maybe my last messages didn’t go through. I’ll send a followup shortly. I talked to engineering and we are confident that the 
Service 'EventsWorker' failed to process item
error is unrelated to flows stuck in Pending. The reason why is that the events worker is only in charge of sending events to Prefect. Flow run states do not depend on events. Most likely there is another cause.
@Moe Are there any other stack traces in your agent or worker logs when flows get stuck in Pending? You should see a few messages related to the flow run being submitted. If you search the logs for the flow run ID you should see more info.
m

Moe

08/23/2023, 4:05 PM
@Emil Christensen Mine don't get stuck in pending. This error pops up in my local terminal that I'm running from. But flow continues to execute and completes successfully. But multiple of these messages pop up while it's running...
Powered by