< Marvin> when prefect 3 self hosted is under load I m seein Prefect Community #ask-marvin

<@ULVA73B9P> when prefect 3 (self hosted) is under...

Janos Krnak

10/10/2025, 1:57 PM

@Marvin when prefect 3 (self hosted) is under load I'm seeing a lot of this errors and I'm not sure where it is originating from. What do you think?

Copy code

RuntimeError: Unexpected ASGI message 'websocket.send', after sending 'websocket.close' or response already completed.
raise RuntimeError(msg % message_type)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 360, in asgi_send
await send(message)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/_exception_handler.py", line 39, in sender
await send(message)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/_exception_handler.py", line 39, in sender
await self._send(message)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/websockets.py", line 86, in send
await self.send({"type": "websocket.send", "text": text})
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/websockets.py", line 176, in send_json
await websocket.send_json(
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/server/api/events.py", line 156, in stream_workspace_events_out
await dependant.call(**solved_result.values)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/fastapi/routing.py", line 390, in app
await func(session)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/routing.py", line 95, in app
await app(scope, receive, sender)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
raise exc
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await wrap_app_handling_exceptions(app, session)(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/routing.py", line 97, in app
await <http://self.app|self.app>(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/routing.py", line 364, in handle
await route.handle(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
await self.middleware_stack(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await app(scope, receive, sender)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
raise exc
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await wrap_app_handling_exceptions(<http://self.app|self.app>, conn)(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await <http://self.app|self.app>(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 19, in __call__
await <http://self.app|self.app>(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/middleware/errors.py", line 151, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
await super().__call__(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
await <http://self.app|self.app>(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle
await route.handle(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
await self.middleware_stack(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await app(scope, receive, sender)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
raise exc
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await wrap_app_handling_exceptions(<http://self.app|self.app>, conn)(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await <http://self.app|self.app>(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/middleware/cors.py", line 77, in __call__
await <http://self.app|self.app>(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/middleware/errors.py", line 151, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
await super().__call__(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
return await <http://self.app|self.app>(scope, receive, send)
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
result = await <http://self.app|self.app>(self.scope, self.asgi_receive, self.asgi_send) # type: ignore[func-returns-value]
File "/opt/conda/envs/prefect/lib/python3.12/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 244, in run_asgi
Traceback (most recent call last):
ERROR: Exception in ASGI application

Marvin

10/10/2025, 1:57 PM

✅ thought for 121.8 seconds

Marvin

10/10/2025, 1:59 PM

Short answer: this is coming from the Prefect Server’s websocket endpoint that streams workspace events to clients (the UI and any subscribers). Under load, a client/proxy is closing the socket and the server is still trying to send an event; Starlette/uvicorn then raises “Unexpected ASGI message 'websocket.send' after sending 'websocket.close'”. Why it happens - In Prefect 3.x, the events endpoint (prefect.server.api.events.stream_workspace_events_out) pushes real-time updates over websockets. When a client disconnects (tab closed, network blip, or reverse proxy idle timeout) and the server tries to send another message, Starlette will raise this error. - Heavier event volume and slower clients make the “send after close” race more likely, so you’ll see more of these under load. - This tends to be noisy, not catastrophic; flows continue to run. You may notice UI real-time panels stopping updates when it occurs. Where it’s originating - The stack points to

prefect.server.api.events.stream_workspace_events_out

, which matches the websocket stream for workspace events. - You can see the events API surface in Prefect’s server module offerings. The general server/event concepts are here: - Prefect 3 Server concepts - Subscriptions/events utilities API What to do next (mitigations and checks) 1) Confirm you’re on the latest Prefect 3.x image. We regularly update the server and its dependencies (Starlette/Uvicorn/FastAPI). If you can, upgrade and retest. 2) Check your reverse proxy/load balancer timeouts and websocket settings: - Nginx example:

Copy code

proxy_http_version 1.1;
     proxy_set_header Upgrade $http_upgrade;
     proxy_set_header Connection "upgrade";
     proxy_read_timeout 3600s;
     proxy_send_timeout 3600s;

- ALB/NLB: increase idle timeout (e.g., 600–1200s) and ensure websocket upgrade is allowed. 3) Reduce churn on websocket connections where possible: - Fewer concurrent UI tabs when load is high - Ensure clients are not rapidly connecting/disconnecting 4) Monitor for symptoms beyond log noise: - Are you seeing missing/lagging real-time updates in the UI? - Any repeated socket disconnects/ping timeouts in your proxy logs? If this persists after updating and adjusting timeouts, it might indicate the server isn’t gracefully handling a specific disconnect path in that endpoint. I can help file an issue with the Prefect team—could you share: - Prefect Server version (image tag) and deployment details (Docker/Kubernetes) - Any reverse proxy in front (Nginx/ALB/etc.) and relevant timeout settings - Rough load profile when this occurs (number of concurrent UI sessions, events/sec) - Whether you observe functional impact (UI real-time panels stalling) or just error logs Happy to dig in further with those details.

Open in Slack

Previous Next