Cloud customer here - we're no longer able to see Tasks in the UI when running flows in kubernetes pods on GKE. Looks like failures with the EventWorker.

"18:54:15.740 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 6 pending items."

Prefect-Worker chart version:

2024.10.15184517

Prefect client on docker images with flows:

prefect = ">=3.0.0rc1"

Task Events in Cloud UI are showing up when running a flow locally. Anyone have similar behavior?

was this only after moving to 3.0?

Cant be 100% but pretty sure @Kevin Grismore I recall having GlobalEventLoopThread errors before switching but at least I was seeing tasks in UI

is there a proxy or firewall rule or something similar that could be blocking websocket connections?

Yes definitely

in 3.x task runs come across as events over websockets rather than http requests

So we probably just need to whitelist or add some NetworkPolicy to the worker pod for our

PREFECT_API_URL?

<wss://api.prefect.cloud/api/accounts/<account>/workspaces/<workspace>>

I would at least give that a shot