Tom Han
10/09/2025, 1:22 AM01:20:41.691 | WARNING | prefect.events.clients - Unable to connect to '<ws://10.1.37.163:4200/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: timed out during opening handshake. Set PREFECT_DEBUG_MODE=1 to see the full error.
01:20:41.693 | ERROR | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 1 pending items.
+ Exception Group Traceback (most recent call last): But I was able to open websocket connection to my host mac mini before. Could it just be a timeout?Marvin
10/09/2025, 1:23 AMTom Han
10/09/2025, 1:24 AM01:20:31.087 | INFO | Flow run 'translucent-vicugna' - > Running pull_with_block step...
01:20:32.832 | INFO | prefect.deployment - Pulled code using block 'github-repository/hcm-core-repo' into 'github-repository-hcm-core-repo'
01:20:41.690 | WARNING | prefect.events.clients - Unable to connect to '<ws://10.1.37.163:4200/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: timed out during opening handshake. Set PREFECT_DEBUG_MODE=1 to see the full error.
01:20:41.691 | ERROR | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 1 pending items.
01:20:46.581 | INFO | Flow run 'translucent-vicugna' - Beginning flow run 'translucent-vicugna' for flow 'inference'
Does this mean that it recovered?Marvin
10/09/2025, 1:24 AMMarvin
10/09/2025, 1:27 AMprefect config view | grep PREFECT_API_URL
Make sure it matches how your server is exposed (http vs https). If the API is https, the WS must be wss.
- From the same environment as your worker/flow, test reachability
nc -vz 10.1.37.163 4200
curl -i <http://10.1.37.163:4200/api/health>
If you’re in Docker on a Mac, try the host alias instead:
export PREFECT_API_URL=<http://host.docker.internal:4200>
- Try a websocket handshake test client (if you have one)
websocat <ws://10.1.37.163:4200/api/events/in>
If this stalls or errors, the path is blocked.
- Check for proxies/VPNs. Proxies often break WS. If you must use a proxy, add NO_PROXY for your API host:
export NO_PROXY=10.1.37.163,localhost,127.0.0.1
export no_proxy=$NO_PROXY
Then re-run your flow/worker from that shell.
- If you’re behind a reverse proxy (e.g., Nginx/Ingress), ensure WS upgrade headers and timeouts are set:
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 600s;
proxy_send_timeout 600s;
A missing upgrade will cause timeouts during the opening handshake.
- Enable debug to see the exact cause
export PREFECT_DEBUG_MODE=1
Then share the expanded error; it often reveals a proxy, TLS, or routing hint.
Notes:
- The endpoint Prefect uses is ws(s)://<your-api>/api/events/in. If your API is only reachable over HTTPS, you must use wss and the PREFECT_API_URL should be https://...
- Your flows can still run, but missed WS means event data (task run details, automations) may not be recorded.
If you can share a bit more:
- Where is the worker/flow running (host, Docker, Kubernetes)?
- Your PREFECT_API_URL value
- Whether there’s a corporate proxy/VPN or reverse proxy in front of the API
- Whether the error is intermittent or constant
Related docs you can reference:
- Server (Helm) and ingress notes – websocket support
- Troubleshooting networking (Cloud, but proxy/NO_PROXY guidance applies)Marvin
10/09/2025, 1:27 AM<ws://10.1.37.163:4200/api/events/in>. That affects event streaming (e.g., real‑time logs/observability), not the execution of the flow itself.
- “Service 'EventsWorker' failed with 1 pending items” means at least one event could not be sent and was dropped.
- “Beginning flow run …” shows the flow execution proceeded anyway. So the run didn’t crash, but the events channel wasn’t connected at that moment.
How to tell if it recovered:
- If it recovers, you’ll typically stop seeing the websocket warnings and you’ll see logs/events in the UI again. With debug on, you may see a message indicating a successful connection.
- Turn on extra detail to confirm: set PREFECT_DEBUG_MODE=1 (as the log suggests) and/or PREFECT_LOGGING_LEVEL=DEBUG to get fuller messages about retries/reconnections.
What to check to fix the websocket:
- Network path from where the flow runs to `http://10.1.37.163:4200`:
- Firewalls/security groups allowing port 4200
- Proxies/load balancers/ingress configured to allow WebSocket upgrades (e.g., NGINX must pass `Upgrade`/`Connection: upgrade`)
- Make sure PREFECT_API_URL is correct and reachable from the runtime. If your API is HTTPS, the events URL should be wss://... (Prefect derives this automatically from PREFECT_API_URL).
- If you’re self‑hosting behind an ingress/Helm, validate the service/ingress supports websockets and the events endpoint.
- For worker/process resiliency, healthchecks can help keep processes alive if there are transient issues.
Helpful references:
- Worker healthchecks
- Prefect Server Helm
If you can share a few more lines after those logs, I can confirm whether the EventsWorker reconnected or if it’s still failing. Are you running a self‑hosted API at 10.1.37.163 behind a proxy or firewall?