https://prefect.io
Join Slack
<@ULVA73B9P> We use prefect 3.6.15. We get this error on some flow runs, it is not systematic. It e...
# ask-community
p
@Marvin We use prefect 3.6.15. We get this error on some flow runs, it is not systematic. It ends up cancelled by the timeout we set. 072217.894 | INFO | Task run 'download_modis_data-d16' - Starting to download the reference period 072217.948 | DEBUG | prefect.events.clients - Reconnecting websocket connection. 072217.949 | DEBUG | prefect.events.clients - Opening websocket connection. 072218.089 | DEBUG | prefect.events.clients - Pinging to ensure websocket connected. 072218.183 | DEBUG | prefect.events.clients - Pong received. Websocket connected. 072218.183 | DEBUG | prefect.events.clients - Resending 0 unconfirmed events. 072218.184 | DEBUG | prefect.events.clients - Finished resending unconfirmed events. 072218.184 | DEBUG | prefect.client - Connecting to API at https://api.prefect.cloud/api/accounts/XXXXXXXXXXXXXXXX/workspaces/XXXXXXXXXXXXXXXX/ 072218.392 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emitting event id=019c6a7a-7b97-7738-822d-37adb06f26c1. 072218.392 | DEBUG | prefect.events.clients - Added event id=019c6a7a-7b97-7738-822d-37adb06f26c1 to unconfirmed events list. There are now 1 unconfirmed events. 072218.392 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emit reconnection attempt 0. 072218.393 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Sending event id=019c6a7a-7b97-7738-822d-37adb06f26c1. 072218.393 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Checkpointing event id=019c6a7a-7b97-7738-822d-37adb06f26c1. 072218.394 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emitting event id=019c6a7a-7ba3-7b01-870f-0d15146b622c. 072218.395 | DEBUG | prefect.events.clients - Added event id=019c6a7a-7ba3-7b01-870f-0d15146b622c to unconfirmed events list. There are now 2 unconfirmed events. 072218.395 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emit reconnection attempt 0. 072218.395 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Sending event id=019c6a7a-7ba3-7b01-870f-0d15146b622c. 072218.395 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Checkpointing event id=019c6a7a-7ba3-7b01-870f-0d15146b622c. 072218.398 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emitting event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d. 072218.398 | DEBUG | prefect.events.clients - Added event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d to unconfirmed events list. There are now 3 unconfirmed events. 072218.398 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emit reconnection attempt 0. 072218.399 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Sending event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d. 072218.399 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Checkpointing event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d. 072256.125 | DEBUG | prefect.events.clients - Retryable error with 1/10 attempts 072256.126 | DEBUG | prefect.events.clients - Reconnecting... 072256.251 | DEBUG | prefect.events.clients - pinging... 072256.346 | DEBUG | prefect.events.clients - authenticating... 072256.460 | DEBUG | prefect.events.clients - auth result {'type': 'auth_success'} 072256.462 | DEBUG | prefect.events.clients - filtering events since 2026-02-17 072156.461702+00:00... [8 other attemps] 084453.314 | DEBUG | prefect.events.clients - Retryable error with 10/10 attempts 084454.315 | DEBUG | prefect.events.clients - Reconnecting... 084454.483 | DEBUG | prefect.events.clients - pinging... 084454.595 | DEBUG | prefect.events.clients - authenticating... 084454.783 | DEBUG | prefect.events.clients - auth result {'type': 'auth_success'} 084454.784 | DEBUG | prefect.events.clients - filtering events since 2026-02-17 084354.784168+00:00... 085454.436 | DEBUG | prefect.events.clients - Retryable error with 11/10 attempts 085454.437 | WARNING | prefect.FlowRunCancellingObserver - The FlowRunCancellingObserver websocket failed with an exception. Switching to polling mode. Traceback (most recent call last): File "/opt/app/venv/lib/python3.11/site-packages/prefect/_observers.py", line 95, in _consume_events async for event in self._events_subscriber: File "/opt/app/venv/lib/python3.11/site-packages/prefect/events/clients.py", line 761, in anext message = orjson.loads(await self._websocket.recv()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app/venv/lib/python3.11/site-packages/websockets/asyncio/connection.py", line 324, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.ConnectionClosedError: no close frame received or sent 085454.440 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of '_check_for_cancelled_flow_runs' 085454.440 | DEBUG | prefect.FlowRunCancellingObserver - Checking for cancelled flow runs 085503.434 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of '_check_for_cancelled_flow_runs' 085503.435 | DEBUG | prefect.FlowRunCancellingObserver - Checking for cancelled flow runs ... same same every ~10s 27 min later: 092213.063 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of '_check_for_cancelled_flow_runs' 092213.064 | DEBUG | prefect.FlowRunCancellingObserver - Checking for cancelled flow runs 092217.808 | ERROR | Task run 'download_modis_data-d16' - Crash detected! Execution was interrupted by an unexpected exception: CancelledError: we think that it may be resolved by 3.6.17, but we are not sure. What do you think ?
m
Your message was too long, here's your message at the allowed limit: We use prefect 3 6 15 We get this error on some flow runs it is not systematic It ends up cancelled by the timeout we set 07 22 17 894 | INFO | Task run 'download_modis_data-d16' - Starting to download the reference period 07 22 17 948 | DEBUG | prefect events clients - Reconnecting websocket connection 07 22 17 949 | DEBUG | prefect events clients - Opening websocket connection 07 22 18 089 | DEBUG | prefect events clients - Pinging to ensure websocket connected 07 22 18 183 | DEBUG | prefect events clients - Pong received Websocket connected 07 22 18 183 | DEBUG | prefect events clients - Resending 0 unconfirmed events 07 22 18 184 | DEBUG | prefect events clients - Finished resending unconfirmed events 07 22 18 184 | DEBUG | prefect client - Connecting to API at <https //api prefect cloud/api/accounts/XXXXXXXXXXXXXXXX/workspaces/XXXXXXXXXXXXXXXX/> 07 22 18 392 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Emitting event id=019c6a7a-7b97-7738-822d-37adb06f26c1 07 22 18 392 | DEBUG | prefect events clients - Added event id=019c6a7a-7b97-7738-822d-37adb06f26c1 to unconfirmed events list There are now 1 unconfirmed events 07 22 18 392 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Emit reconnection attempt 0 07 22 18 393 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Sending event id=019c6a7a-7b97-7738-822d-37adb06f26c1 07 22 18 393 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Checkpointing event id=019c6a7a-7b97-7738-822d-37adb06f26c1 07 22 18 394 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Emitting event id=019c6a7a-7ba3-7b01-870f-0d15146b622c 07 22 18 395 | DEBUG | prefect events clients - Added event id=019c6a7a-7ba3-7b01-870f-0d15146b622c to unconfirmed events list There are now 2 unconfirmed events 07 22 18 395 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Emit reconnection attempt 0 07 22 18 395 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Sending event id=019c6a7a-7ba3-7b01-870f-0d15146b622c 07 22 18 395 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Checkpointing event id=019c6a7a-7ba3-7b01-870f-0d15146b622c 07 22 18 398 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Emitting event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d 07 22 18 398 | DEBUG | prefect events clients - Added event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d to unconfirmed events list There are now 3 unconfirmed events 07 22 18 398 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Emit reconnection attempt 0 07 22 18 399 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Sending event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d 07 22 18 399 | DEBUG | prefect events clients - EventsClient(id=128014307590800) Checkpointing event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d 07 22 56 125 | DEBUG | prefect events clients - Retryable error with 1/10 attempts 07 22 56 126 | DEBUG | prefect events clients - Reconnecting 07 22 56 251 | DEBUG | prefect events clients - pinging 07 22 56 346 | DEBUG | prefect events clients - authenticating 07 22 56 460 | DEBUG | prefect events clients - auth result {'type' 'auth_success'} 07 22 56 462 | DEBUG | prefect events clients - filtering events since 2026-02-17 07 21 56 461702+00 00 [8 other attemps] 08 44 53 314
p
@Marvin We use prefect 3.6.15. We get this error on some flow runs, it is not systematic. It ends up cancelled by the timeout we set. 072217.894 | INFO | Task run 'download_modis_data-d16' - Starting to download the reference period 072217.948 | DEBUG | prefect.events.clients - Reconnecting websocket connection. 072217.949 | DEBUG | prefect.events.clients - Opening websocket connection. 072218.089 | DEBUG | prefect.events.clients - Pinging to ensure websocket connected. 072218.183 | DEBUG | prefect.events.clients - Pong received. Websocket connected. 072218.183 | DEBUG | prefect.events.clients - Resending 0 unconfirmed events. 072218.184 | DEBUG | prefect.events.clients - Finished resending unconfirmed events. 072218.184 | DEBUG | prefect.client - Connecting to API at https://api.prefect.cloud/api/accounts/XXXXXXXXXXXXXXXX/workspaces/XXXXXXXXXXXXXXXX/ 072218.392 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emitting event id=019c6a7a-7b97-7738-822d-37adb06f26c1. 072218.392 | DEBUG | prefect.events.clients - Added event id=019c6a7a-7b97-7738-822d-37adb06f26c1 to unconfirmed events list. There are now 1 unconfirmed events. 072218.392 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emit reconnection attempt 0. 072218.393 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Sending event id=019c6a7a-7b97-7738-822d-37adb06f26c1. 072218.393 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Checkpointing event id=019c6a7a-7b97-7738-822d-37adb06f26c1. 072218.394 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emitting event id=019c6a7a-7ba3-7b01-870f-0d15146b622c. 072218.395 | DEBUG | prefect.events.clients - Added event id=019c6a7a-7ba3-7b01-870f-0d15146b622c to unconfirmed events list. There are now 2 unconfirmed events. 072218.395 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emit reconnection attempt 0. 072218.395 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Sending event id=019c6a7a-7ba3-7b01-870f-0d15146b622c. 072218.395 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Checkpointing event id=019c6a7a-7ba3-7b01-870f-0d15146b622c. 072218.398 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emitting event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d. 072218.398 | DEBUG | prefect.events.clients - Added event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d to unconfirmed events list. There are now 3 unconfirmed events. 072218.398 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Emit reconnection attempt 0. 072218.399 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Sending event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d. 072218.399 | DEBUG | prefect.events.clients - EventsClient(id=128014307590800): Checkpointing event id=019c6a7a-7c94-795d-bf48-4f8aed623a6d. 072256.125 | DEBUG | prefect.events.clients - Retryable error with 1/10 attempts 072256.126 | DEBUG | prefect.events.clients - Reconnecting... 072256.251 | DEBUG | prefect.events.clients - pinging... 072256.346 | DEBUG | prefect.events.clients - authenticating... 072256.460 | DEBUG | prefect.events.clients - auth result {'type': 'auth_success'} 072256.462 | DEBUG | prefect.events.clients - filtering events since 2026-02-17 072156.461702+00:00... [8 other attempts]
m
✅ thought for 295.5 seconds
Thanks for sharing the logs — that helps. What you’re seeing is the events websocket client/subscriber reconnecting in the background. The “Reconnecting… pinging… authenticating… filtering events since …” messages are from Prefect’s events system re-establishing its websocket and replaying missed events. This runs on a background worker and, in Prefect 3.x, should not block your task code from executing. A few concrete points and next steps: - Why those logs appear - Prefect emits events (e.g., task/flow state changes) and also subscribes to certain events (e.g., cancellations) over websockets. - Transient network blips or load balancers that close idle connections will trigger the reconnect loop you’re seeing. The subscriber will “filter events since …” to catch up on anything it may have missed during the disconnect. - Known fix in newer Prefect versions - There was a bug around websocket timeouts when watching long-running flow runs that was fixed after 3.6.15. It’s called out in the release notes as “Fix websocket timeout when watching long-running flow runs.” - Recommendation: upgrade to 3.6.17 or newer, which includes this fix. - Release notes: Prefect 3.6.17 release notes - PR reference: Fix websocket timeout when watching long-running flow runs (#20590) - Settings you can tune - The events client exposes 
reconnection_attempts
(default 10) and 
checkpoint_every
(default 700) on 
PrefectEventsClient
, but it’s typically managed internally. There isn’t a separate environment variable for websocket timeout tuning. - From our API tree:
Copy code
PrefectEventsClient(api_url: Optional[str]=None, reconnection_attempts: int=10, checkpoint_every: int=700)
- Why your runs are timing out - Given the above, the reconnect loop itself should not stall task execution. If your flow ends “Cancelled by timeout,” it usually means your task or flow hit its own timeout while doing work (e.g., a network call that never returned). - If the task is downloading data, ensure your HTTP client enforces explicit connect/read timeouts and, if possible, retries/backoff. For example with requests:
Copy code
r = requests.get(url, timeout=(10, 120))  # 10s connect, 120s read
or with httpx:
Copy code
timeout = httpx.Timeout(connect=10.0, read=120.0, write=120.0, pool=60.0)
    r = httpx.get(url, timeout=timeout)
- Infra/network considerations - If you run behind a proxy/LB with short idle timeouts, frequent websocket reconnects are expected. That’s okay; they’re background. If you can, increase idle timeouts or enable keep-alives. But this should not block your task. Questions to help us zero in: - What infrastructure are these runs on (process worker, Docker, Kubernetes)? - Is the timeout on the task, the flow, or both — and what’s the value? - Do you see the task code doing a long network call without timeouts when the hang occurs? - After upgrading to 3.6.17+, do the intermittent hangs still appear? If you can share a flow run ID and timestamp where this happened, I can take a closer look at the surrounding runtime signals.
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by