https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> i am deploying prefect on my openshif...
# ask-marvin
a
@Marvin i am deploying prefect on my openshift tool on my own server and when i want to create the worker i get this error
Copy code
Discovered type 'process' for work pool 'openshift-pool'.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/cli/_types.py", line 159, in sync_fn
    return asyncio.run(async_fn(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/cli/worker.py", line 168, in start
    await worker.start(
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/process.py", line 158, in start
    async with self as worker:
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/process.py", line 314, in __aenter__
    self._runner = await self._exit_stack.enter_async_context(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/contextlib.py", line 650, in enter_async_context
    result = await _enter(cm)
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/runner/runner.py", line 1571, in __aenter__
    self._cancelling_observer = await self._exit_stack.enter_async_context(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/contextlib.py", line 650, in enter_async_context
    result = await _enter(cm)
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/runner/_observers.py", line 157, in __aenter__
    self._events_subscriber = await self._exit_stack.enter_async_context(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/contextlib.py", line 650, in enter_async_context
    result = await _enter(cm)
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/events/clients.py", line 555, in __aenter__
    await self._reconnect()
  File "/usr/local/lib/python3.11/site-packages/prefect/events/clients.py", line 566, in _reconnect
    self._websocket = await self._connect.__aenter__()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/client.py", line 587, in __aenter__
    return await self
           ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/client.py", line 543, in __await_impl__
    await self.connection.handshake(
  File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/client.py", line 114, in handshake
    raise self.protocol.handshake_exc
  File "/usr/local/lib/python3.11/site-packages/websockets/client.py", line 325, in parse
    self.process_response(response)
  File "/usr/local/lib/python3.11/site-packages/websockets/client.py", line 142, in process_response
    raise InvalidStatus(response)
websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 500
An exception occurred.
how can i fix it what are the possible issues here ?
m
✅ thought for 336.9 seconds
Thanks for sharing the traceback — that failure happens while the worker is trying to open a WebSocket to the Prefect API for events/cancellation. In self‑hosted setups (especially behind OpenShift Routes/ingress), HTTP 500 during the WebSocket handshake is almost always a proxy/route config issue or a mismatch in auth/URL. Here’s how to fix it and what to check. Quick triage checklist - Confirm your API URL is correct and includes /api. - Example: 
PREFECT_API_URL=<https://prefect.example.com/api>
- Make sure your reverse proxy/route allows WebSocket upgrades and forwards the right headers. - Must forward 
Upgrade
, 
Connection
, 
Host
, 
X-Forwarded-Proto
, and preserve 
Sec-WebSocket-Protocol: prefect
- Check TLS/termination: - If you terminate TLS at the OpenShift Route (edge/re-encrypt), make sure the worker connects via 
https://
and your backend service is reachable over HTTP by the router. - If using a custom CA, ensure the worker trusts it (system trust store or set `SSL_CERT_FILE`/`REQUESTS_CA_BUNDLE`). - If you set API auth on the server, set the matching client auth on the worker: - Server: 
PREFECT_SERVER_API_AUTH_STRING
- Worker: 
PREFECT_API_AUTH_STRING
(must match exactly) - Get the server logs around the time of the 500 — the exception will usually be logged there and will pinpoint the root cause. OpenShift/Ingress specifics - OpenShift Routes support WebSockets, but you typically need longer timeouts for long‑lived connections:
Copy code
apiVersion: <http://route.openshift.io/v1|route.openshift.io/v1>
kind: Route
metadata:
  name: prefect-api
  annotations:
    <http://haproxy.router.openshift.io/timeout|haproxy.router.openshift.io/timeout>: 1h
spec:
  host: <http://prefect.example.com|prefect.example.com>
  to:
    kind: Service
    name: prefect-api
  port:
    targetPort: 4200
  tls:
    termination: edge   # or reencrypt/passthrough as appropriate
    insecureEdgeTerminationPolicy: Redirect
- If you use another proxy in front (NGINX/HAProxy), ensure WebSocket headers are forwarded and HTTP/1.1 is used. For example (NGINX):
Copy code
location /api {
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";

  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  # Preserve auth if you use it
  proxy_set_header Authorization $http_authorization;
  proxy_pass_header Authorization;

  proxy_buffering off;
  proxy_read_timeout 3600s;
  proxy_send_timeout 3600s;

  proxy_pass <http://prefect-backend>;  # your service
}
Why this fails with HTTP 500 - Missing/stripped WebSocket headers (most commonly 
Upgrade
, 
Connection
, or 
Sec-WebSocket-Protocol: prefect
) cause the backend to error during handshake. - Path rewriting that drops the /api prefix can route to the wrong handler. - TLS/host mismatch or untrusted certs can break the connection. - If server API auth is enabled but the worker didn’t send the matching token, the server typically returns 403 — but some proxy chains can surface this as 500. How to validate quickly 1) Verify the API health:
Copy code
curl -i <https://prefect.example.com/api/health>
2) Verify WS upgrade reaches the backend (expect 101 Switching Protocols, not 500):
Copy code
curl -i -N \
  -H "Connection: Upgrade" \
  -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
  -H "Sec-WebSocket-Version: 13" \
  -H "Sec-WebSocket-Protocol: prefect" \
  <https://prefect.example.com/api/events/out>
3) Try bypassing the Route temporarily (to isolate the router): - From a pod in the same namespace, set 
PREFECT_API_URL
to the internal service URL, e.g. 
<http://prefect-api:4200/api>
, then start the worker. If it works, the Route/ingress is the culprit. Notes about disabling events - There isn’t a supported environment variable to disable the worker’s event/cancellation WebSocket in Prefect 3.x. If the WS cannot be established, the worker can’t finish starting. The fix is to ensure the route/proxy allows the WS handshake.
If you share a bit more, I can be more specific: - Your Prefect version (
prefect version
) - The exact 
PREFECT_API_URL
your worker is using - OpenShift Route YAML (or any additional proxy config in front) - Whether you’ve set 
PREFECT_SERVER_API_AUTH_STRING
on the server and 
PREFECT_API_AUTH_STRING
on the worker - TLS termination mode (edge/reencrypt/passthrough) and whether you use a custom CA With that, I can help you pinpoint the exact header/route change needed.
20 Views
Open in Slack
PreviousNext
Powered by