https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> why i am getting this error ```2025-0...
# ask-marvin
r
@Marvin why i am getting this error
Copy code
2025-09-24T22:55:37.193+05:30
17:25:37.189 | ERROR   | prefect.FlowRunCancellingObserver - Error consuming events

17:25:37.189 | ERROR | prefect.FlowRunCancellingObserver - Error consuming events
2025-09-24T22:55:37.193+05:30
Traceback (most recent call last):

Traceback (most recent call last):
2025-09-24T22:55:37.193+05:30
  File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/client.py", line 541, in __await_impl__

File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/client.py", line 541, in __await_impl__
2025-09-24T22:55:37.193+05:30
    self.connection = await self.create_connection()

self.connection = await self.create_connection()
2025-09-24T22:55:37.193+05:30
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-24T22:55:37.193+05:30
  File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/client.py", line 467, in create_connection

File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/client.py", line 467, in create_connection
2025-09-24T22:55:37.193+05:30
    _, connection = await loop.create_connection(factory, **kwargs)

_, connection = await loop.create_connection(factory, **kwargs)
2025-09-24T22:55:37.193+05:30
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-24T22:55:37.193+05:30
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1046, in create_connection
m
✅ thought for 160.6 seconds
Thanks for sharing the snippet — that error is coming from Prefect’s FlowRunCancellingObserver. In Prefect 3.x, when you cancel a flow run, the runner opens a WebSocket connection to the API to “listen” for cancellation-related events. Your traceback shows that the WebSocket could not be created, so the observer logs “Error consuming events” with a stack trace from 
websockets/asyncio/...create_connection()
. Common causes: - Network/proxy/firewall blocking outbound WebSocket (wss) traffic from the worker/runner to your Prefect API - Reverse proxy/load balancer in front of a self-hosted Prefect server not allowing WebSocket upgrades - TLS/hostname/cert mismatch for the API URL (e.g., 
PREFECT_API_URL
points to https but the proxy only supports ws, or cert issues) - A bug in certain Prefect 3.x versions where the observer didn’t gracefully fall back when the socket fails Good news: we’ve added a fallback to polling when the WebSocket cannot be established so cancellations still succeed even if the socket drops. That change is here: Add polling fallback for FlowRunCancellingObserver. If you’re not on the latest 3.x, please upgrade — that alone often resolves this exact log/noise and stuck-cancelling behavior. Related context: Flows stuck in Cancelling state when ran via work pools. To help pinpoint your case, could you share: - Prefect version (
prefect version
) - Cloud or self-hosted? If self-hosted, anything fronting it (nginx/ALB/ingress) - The value of 
PREFECT_API_URL
on the worker/runner - Worker type/runtime (process, docker, k8s, ECS, etc.) - Whether you’re behind an HTTP(S) proxy; if so, do you have 
NO_PROXY
set for your API host? - The bottom lines of the traceback (the actual exception like timeout, connection refused, certificate error, etc.) Quick checks you can do: - Upgrade to the latest Prefect 3.x and retry a cancel - Ensure outbound wss:// traffic to your API host:443 is allowed from the worker - If self-hosted, confirm your proxy supports WebSocket upgrade (Connection: Upgrade / Upgrade: websocket) to the Prefect API’s events endpoint - Double-check 
PREFECT_API_URL
is correct (https vs http will determine wss vs ws) If you want, I can help you test WebSocket connectivity from the worker container/host with a short snippet — just let me know your environment (Cloud vs Server) and I’ll tailor it.
Open in Slack
PreviousNext
Powered by