<@ULVA73B9P> I kept getting ` 01:20:41.691 | WARNI...
# ask-marvin
t
@Marvin I kept getting
Copy code
01:20:41.691 | WARNING | prefect.events.clients - Unable to connect to '<ws://10.1.37.163:4200/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: timed out during opening handshake. Set PREFECT_DEBUG_MODE=1 to see the full error.
01:20:41.693 | ERROR | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 1 pending items.
+ Exception Group Traceback (most recent call last):
But I was able to open websocket connection to my host mac mini before. Could it just be a timeout?
m
thought for 245.0 seconds
t
@Marvin
Copy code
01:20:31.087 | INFO | Flow run 'translucent-vicugna' - > Running pull_with_block step...
01:20:32.832 | INFO | prefect.deployment - Pulled code using block 'github-repository/hcm-core-repo' into 'github-repository-hcm-core-repo'
01:20:41.690 | WARNING | prefect.events.clients - Unable to connect to '<ws://10.1.37.163:4200/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: timed out during opening handshake. Set PREFECT_DEBUG_MODE=1 to see the full error.
01:20:41.691 | ERROR | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 1 pending items.
01:20:46.581 | INFO | Flow run 'translucent-vicugna' - Beginning flow run 'translucent-vicugna' for flow 'inference'
Does this mean that it recovered?
m
thought for 200.5 seconds
Short answer: yes, that message means the WebSocket handshake to your Prefect API timed out. In practice, persistent handshaking timeouts are almost always a network-path problem (proxy, firewall, reverse proxy not upgrading to websockets, wrong host/port, container-to-host routing), not just a “slow server”. Quick checks to pinpoint it: - Confirm the API URL your runs are using
Copy code
prefect config view | grep PREFECT_API_URL
Make sure it matches how your server is exposed (http vs https). If the API is https, the WS must be wss. - From the same environment as your worker/flow, test reachability
Copy code
nc -vz 10.1.37.163 4200
  curl -i <http://10.1.37.163:4200/api/health>
If you’re in Docker on a Mac, try the host alias instead:
Copy code
export PREFECT_API_URL=<http://host.docker.internal:4200>
- Try a websocket handshake test client (if you have one)
Copy code
websocat <ws://10.1.37.163:4200/api/events/in>
If this stalls or errors, the path is blocked. - Check for proxies/VPNs. Proxies often break WS. If you must use a proxy, add NO_PROXY for your API host:
Copy code
export NO_PROXY=10.1.37.163,localhost,127.0.0.1
  export no_proxy=$NO_PROXY
Then re-run your flow/worker from that shell. - If you’re behind a reverse proxy (e.g., Nginx/Ingress), ensure WS upgrade headers and timeouts are set:
Copy code
proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_read_timeout 600s;
  proxy_send_timeout 600s;
A missing upgrade will cause timeouts during the opening handshake. - Enable debug to see the exact cause
Copy code
export PREFECT_DEBUG_MODE=1
Then share the expanded error; it often reveals a proxy, TLS, or routing hint. Notes: - The endpoint Prefect uses is ws(s)://<your-api>/api/events/in. If your API is only reachable over HTTPS, you must use wss and the
PREFECT_API_URL
should be https://... - Your flows can still run, but missed WS means event data (task run details, automations) may not be recorded. If you can share a bit more: - Where is the worker/flow running (host, Docker, Kubernetes)? - Your
PREFECT_API_URL
value - Whether there’s a corporate proxy/VPN or reverse proxy in front of the API - Whether the error is intermittent or constant Related docs you can reference: - Server (Helm) and ingress notes – websocket support - Troubleshooting networking (Cloud, but proxy/NO_PROXY guidance applies)
Short answer: not exactly. It means your flow kept running, but the events websocket had an issue. What the logs are saying: - The warning/error is from the EventsWorker failing to open the websocket at
<ws://10.1.37.163:4200/api/events/in>
. That affects event streaming (e.g., real‑time logs/observability), not the execution of the flow itself. - “Service 'EventsWorker' failed with 1 pending items” means at least one event could not be sent and was dropped. - “Beginning flow run …” shows the flow execution proceeded anyway. So the run didn’t crash, but the events channel wasn’t connected at that moment. How to tell if it recovered: - If it recovers, you’ll typically stop seeing the websocket warnings and you’ll see logs/events in the UI again. With debug on, you may see a message indicating a successful connection. - Turn on extra detail to confirm: set
PREFECT_DEBUG_MODE=1
(as the log suggests) and/or
PREFECT_LOGGING_LEVEL=DEBUG
to get fuller messages about retries/reconnections. What to check to fix the websocket: - Network path from where the flow runs to `http://10.1.37.163:4200`: - Firewalls/security groups allowing port 4200 - Proxies/load balancers/ingress configured to allow WebSocket upgrades (e.g., NGINX must pass `Upgrade`/`Connection: upgrade`) - Make sure
PREFECT_API_URL
is correct and reachable from the runtime. If your API is HTTPS, the events URL should be
wss://...
(Prefect derives this automatically from
PREFECT_API_URL
). - If you’re self‑hosting behind an ingress/Helm, validate the service/ingress supports websockets and the events endpoint. - For worker/process resiliency, healthchecks can help keep processes alive if there are transient issues. Helpful references: - Worker healthchecks - Prefect Server Helm If you can share a few more lines after those logs, I can confirm whether the EventsWorker reconnected or if it’s still failing. Are you running a self‑hosted API at 10.1.37.163 behind a proxy or firewall?