<@ULVA73B9P> we are getting below error in prefect...
# ask-marvin
m
@Marvin we are getting below error in prefect 3.6.24 server version:
Copy code
prefect.server.utilities.subscriptions - WebSocket connection rejected: 'prefect' subprotocol required when auth is configured
m
thought for 104.1 seconds
Thanks for the details. That log means your WebSocket client connected to the Prefect Server while API auth is enabled, but the
Sec-WebSocket-Protocol
header with value
prefect
did not reach the server. In Prefect 3.x, when
PREFECT_SERVER_API_AUTH_STRING
is set, all WebSocket connections must: - Negotiate the
prefect
subprotocol (send
Sec-WebSocket-Protocol: prefect
in the handshake), and - Send an initial auth message on the socket before any other messages. Most commonly, this happens because a reverse proxy (Nginx/Traefik/etc.) is stripping the
Sec-WebSocket-Protocol
header. What to check/fix - If you’re behind Nginx, make sure you forward the subprotocol and upgrade headers:
Copy code
location /api {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;

    # WebSocket upgrade
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    # IMPORTANT: pass through the subprotocol header
    proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;

    # If you use Authorization for API auth
    proxy_set_header Authorization $http_authorization;
    proxy_pass_header Authorization;

    proxy_pass <http://prefect_api>;
}
- If you’re using Traefik, ensure the subprotocol header is preserved/forwarded. For example (labels):
Copy code
traefik.http.middlewares.websocket.headers.customrequestheaders.Sec-WebSocket-Protocol=prefect
Alternatively, ensure your Traefik config does not drop the
Sec-WebSocket-Protocol
header. - If you have a custom WebSocket client (not the Prefect UI), it must: - Include
Sec-WebSocket-Protocol: prefect
in the handshake, and - Send the first message as:
Copy code
{
  "type": "auth",
  "token": "<your-auth-string>"
}
- Quick test of the handshake (replace URL with your host/path):
Copy code
curl -i -N \
  -H "Sec-WebSocket-Protocol: prefect" \
  -H "Connection: Upgrade" \
  -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Key: test" \
  -H "Sec-WebSocket-Version: 13" \
  <https://your-prefect-host.example.com/api/<ws-endpoint>>
Notes - The Prefect UI and official clients already send the
prefect
subprotocol. If you see this error while loading the UI, it’s almost always a proxy dropping the header. - When auth is not configured, Prefect will accept legacy sockets without the subprotocol; once you enable auth, the subprotocol becomes required. Docs - Reverse proxy guidance and self-hosted notes: Self-hosted reverse proxy configuration If you can share: - Which proxy/load balancer you’re using (Nginx, Traefik, ALB, etc.) - Your proxy config for the
/api
path - Whether
PREFECT_SERVER_API_AUTH_STRING
is set …I can help validate the exact changes needed.
m
@Marvin we are using self hosted server
m
thought for 23.1 seconds
Got it — thanks for confirming you’re self-hosting. In self-hosted setups, this error almost always comes from the reverse proxy in front of Prefect stripping the
Sec-WebSocket-Protocol
header. With API auth enabled (
PREFECT_SERVER_API_AUTH_STRING
), Prefect requires WebSocket clients to negotiate the
prefect
subprotocol. The Prefect UI already sends it — it just needs to reach the API. What to do (pick your proxy) - NGINX (bare metal or ingress) - Ensure you forward both the upgrade headers and the subprotocol header:
Copy code
location /api {
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Real-IP $remote_addr;

      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

      # CRITICAL: pass through the subprotocol header
      proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;

      proxy_pass <http://prefect_api>;
  }
- For Kubernetes NGINX Ingress, add:
Copy code
<http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "3600"
  <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "3600"
and make sure your backend service is mapped to
/api
with the header passthrough (via an NGINX snippet if needed). - Traefik - Preserve the subprotocol header (example with Docker labels):
Copy code
traefik.http.middlewares.prefect-ws.headers.customrequestheaders.Sec-WebSocket-Protocol=prefect
  traefik.http.routers.prefect-api.middlewares=prefect-ws@docker
- Or set an equivalent middleware in your static/dynamic config to forward
Sec-WebSocket-Protocol
. - Other proxies/load balancers (ALB, Caddy, Cloudflare, etc.) - Ensure WebSockets are enabled and both: -
Upgrade
and
Connection: upgrade
-
Sec-WebSocket-Protocol
are passed through unmodified. Quick diagnostics - From your workstation or a pod, hit your API through the proxy and confirm the header makes it through:
Copy code
curl -i -N \
    -H "Sec-WebSocket-Protocol: prefect" \
    -H "Connection: Upgrade" \
    -H "Upgrade: websocket" \
    -H "Sec-WebSocket-Key: test" \
    -H "Sec-WebSocket-Version: 13" \
    https://<your-host>/api/<ws-endpoint>
- In browser devtools (Network tab), check the WebSocket handshake request/response: - Request should include
Sec-WebSocket-Protocol: prefect
- Response should echo
Sec-WebSocket-Protocol: prefect
If the request shows it but Prefect logs that it’s missing, the proxy is stripping it. Reference - Reverse proxy guidance for self-hosted: Self-hosted reverse proxy configuration If you can share: - Which proxy/ingress/LB you’re using - A redacted snippet of your proxy config for the Prefect API route - How you deployed Prefect (Docker, Helm, etc.) …I’ll tailor the exact config you need to resolve this.
m
@Marvin can this be related to version mismatch. Server is on 3.6.24 whereas workpool is on 3.1.8
m
thought for 82.0 seconds
Short answer: This is very unlikely to be a worker/server version mismatch issue. - Workers do not use WebSockets for core scheduling/polling. They talk to the server over HTTP REST (e.g., get scheduled flow runs), so the ‘prefect’ WebSocket subprotocol requirement does not affect normal worker operation. - The ‘prefect’ subprotocol is only relevant to WebSocket endpoints used by the UI and real-time streams (logs/events) and some optional real-time features (e.g., event subscribers, cancellation observer). When
PREFECT_SERVER_API_AUTH_STRING
is set, those WebSocket connections must negotiate
Sec-WebSocket-Protocol: prefect
. What your error points to - A WebSocket client (most often the Prefect UI through your proxy) reached the server without the
Sec-WebSocket-Protocol: prefect
header being preserved. That’s typically a reverse proxy config issue, not a client version mismatch. - Even if a very old client tried to connect without the subprotocol, it would affect only real-time WS features, not worker scheduling. Recommended next steps - Focus on your proxy/ingress configuration to ensure the
Sec-WebSocket-Protocol
header is passed through alongside WebSocket upgrade headers. - If the UI is the trigger, open your browser devtools → Network → WS handshake and verify: - Request has
Sec-WebSocket-Protocol: prefect
- Response echoes
Sec-WebSocket-Protocol: prefect
If the request has it but Prefect logs complain, the proxy is stripping it. If you share which proxy/ingress you’re using and a redacted config snippet for the Prefect API route, I can suggest the exact header settings to fix it.