https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P>, I am running prefect server and work...
# ask-marvin
m
@Marvin, I am running prefect server and worker in the same kubernetes cluster, both version 3.7.5. I am currently seeing the following error log in my prefect server: 
WebSocket connection rejected: 'prefect' subprotocol required when auth is configured.
The worker also has error logs, because of the server rejecting its connection.
m
✅ thought for 152.5 seconds
This usually means auth is enabled on the server, but the worker is not connecting with the expected auth configuration. In Prefect 3.7.5, when 
PREFECT_SERVER_API_AUTH_STRING
is set on the server, WebSocket clients must: 1. connect with the 
prefect
websocket subprotocol, and 2. authenticate with a matching auth token If that auth setup is incomplete or mismatched, the server rejects the worker connection and you’ll see:
Copy code
text
WebSocket connection rejected: 'prefect' subprotocol required when auth is configured.
Most likely fix Make sure these settings are aligned: On the Prefect server:
Copy code
text
PREFECT_SERVER_API_AUTH_STRING=admin:password123
On the worker:
Copy code
text
PREFECT_API_URL=http://<your-prefect-server-service>:4200/api
PREFECT_API_AUTH_STRING=admin:password123
The auth strings must be identical. Also check that the worker does not have a conflicting 
PREFECT_API_KEY
set. In current Prefect behavior, 
PREFECT_API_KEY
can take precedence over 
PREFECT_API_AUTH_STRING
, which can break self-hosted auth unexpectedly. So on the worker, if you are using self-hosted server auth, remove/unset:
Copy code
text
PREFECT_API_KEY
Things to verify 1. Server has auth enabled? - If yes, worker must use 
PREFECT_API_AUTH_STRING
2. Worker points to the right API URL? - Should usually be something like:
Copy code
text
   <http://prefect-server.prefect.svc.cluster.local:4200/api>
3. Worker has no Cloud API key configured? - Remove 
PREFECT_API_KEY
if present 4. Both pods restarted after env var changes? If you are using Helm Typical setup is: Server values:
Copy code
yaml
server:
  basicAuth:
    enabled: true
Worker values:
Copy code
yaml
worker:
  apiConfig: selfHostedServer
  selfHostedServerApiConfig:
    apiUrl: <http://prefect-server.prefect.svc.cluster.local:4200/api>
    basicAuth:
      enabled: true
And both secrets should contain the same 
auth-string
. Relevant docs - Self-hosted Prefect - Security settings - Prefect server Helm guide If you want, paste your server/worker env vars or Helm values (redact secrets), and I can help spot the exact mismatch.
m
@Marvin That does not seem to be the problem. See below my helm-values for both server and worker. Both are deployed in the same namespace and using the same secret for the auth-string
Copy code
#SERVER 
server:
  basicAuth:
    enabled: true
    existingSecret: prefect-basicauth

  uiConfig:
    prefectUiApiUrl: "<REDACTED>"

  extraVolumes:
    - name: postgres-ssl-cert
      secret:
        secretName: postgres-ssl-cert

  extraVolumeMounts:
    - name: postgres-ssl-cert
      mountPath: "/home/prefect/.postgresql/"
      readOnly: true

  resources:
    limits:
      cpu: null

secret:
  create: false
  name: prefect-postgresql-credentials

postgresql:
  enabled: false

# WORKER
worker:
  apiConfig: selfHostedServer
  # Set cluster uid because of: <https://github.com/PrefectHQ/prefect-kubernetes/blob/d1a5643081d428d18d3cd88dcc169b1bfbfa4f96/prefect_kubernetes/worker.py#L934>
  clusterUid: "<REDACTED>"


  resources:
    limits:
      cpu: null
  selfHostedServerApiConfig:
    apiUrl: <http://prefect-server>.{server_namespace}.svc.cluster.local:4200/api
    basicAuth:
      enabled: true
      existingSecret: prefect-basicauth

  config:
    workPool: <REDACTED>

    # Refer to generated configMap
    baseJobTemplate:
      existingConfigMapName: prefect-worker-base-job-template
m
✅ thought for 156.0 seconds
Thanks — that helps. Based on your values, I agree: “same secret is configured” means the obvious mismatch is less likely. The important nuance here is: That server log line does not necessarily mean the failing client is the worker-channel connection. It can come from any Prefect websocket endpoint, including: - worker channel - events stream - log subscriptions - other subscription endpoints So even if the worker pod is configured correctly, a flow-run pod or another Prefect client may be the one getting rejected. What I’d check next: 1. Verify the actual env vars on the running pods, not just Helm values Please inspect all 3 of these: Server pod
Copy code
bash
kubectl exec -n <ns> deploy/prefect-server -- printenv | grep PREFECT_SERVER_API_AUTH_STRING
Worker pod
Copy code
bash
kubectl exec -n <ns> deploy/<worker-deployment-name> -- printenv | grep -E 'PREFECT_API_URL|PREFECT_API_AUTH_STRING|PREFECT_API_KEY'
One flow-run pod created by the worker
Copy code
bash
kubectl exec -n <ns> <flow-run-pod-name> -- printenv | grep -E 'PREFECT_API_URL|PREFECT_API_AUTH_STRING|PREFECT_API_KEY'
Why this matters: - 
worker.selfHostedServerApiConfig.basicAuth.enabled
should configure the worker deployment - but your flow run pods may still be missing 
PREFECT_API_AUTH_STRING
- if the rejected websocket is from the run’s events client, the worker config can be fine while runs still fail 2. Check whether a flow-run pod is the one failing Your server warning is consistent with a Prefect client connecting to 
/api/events/in
without the required websocket subprotocol/auth flow. That often shows up from code running inside a flow-run container, not from the worker’s long-lived control channel. If you can, compare timestamps: - server warning timestamp - worker log timestamp - flow-run pod logs If the warning appears when a flow run starts, that strongly points to the flow-run container rather than the worker itself. 3. Check for 
PREFECT_API_KEY
anywhere Even in self-hosted mode, if 
PREFECT_API_KEY
is present, it can interfere with auth selection. Check both worker and flow-run pods:
Copy code
bash
kubectl exec -n <ns> deploy/<worker-deployment-name> -- printenv | grep PREFECT_API_KEY
kubectl exec -n <ns> <flow-run-pod-name> -- printenv | grep PREFECT_API_KEY
If it exists, remove it. 4. Confirm the problem is not chart-side env injection gap Your values show:
Copy code
yaml
worker:
  selfHostedServerApiConfig:
    basicAuth:
      enabled: true
      existingSecret: prefect-basicauth
That should configure worker auth, but the next question is whether your base job template also injects auth into spawned job pods. You have:
Copy code
yaml
config:
  baseJobTemplate:
    existingConfigMapName: prefect-worker-base-job-template
That’s a strong clue. If that custom base job template overrides the default env injection, the worker pod may be authenticated while spawned flow-run pods are not. So I’d inspect the generated/used base job template ConfigMap:
Copy code
bash
kubectl get configmap -n <ns> prefect-worker-base-job-template -o yaml
Look specifically for whether the job pod template includes: - 
PREFECT_API_URL
- 
PREFECT_API_AUTH_STRING
If those are missing from the spawned job template, that is very likely the root cause. 5. Important likely culprit: custom 
baseJobTemplate
 Given your config, my current best guess is: - worker deployment auth is OK - server auth is OK - spawned Kubernetes job pods from 
baseJobTemplate
are missing self-hosted auth env vars - those pods attempt websocket connections for events/logging - server rejects them and logs:
Copy code
text
  WebSocket connection rejected: 'prefect' subprotocol required when auth is configured.
What I’d like to see next Please share these outputs (with secrets redacted): 1. Worker pod env: ```bash
kubectl exec -n <ns> deploy/<worker-deployment-name> -- printenv | grep -E 'PREFECT_API_URL|PREFECT_API_AUTH_STRING|PREFECT_API_KEY'
Copy code
2. Flow-run pod env:
```bash
kubectl exec -n <ns> <flow-run-pod-name> -- printenv | grep -E 'PREFECT_API_URL|PREFECT_API_AUTH_STRING|PREFECT_API_KEY'
3. Base job template ConfigMap:
Copy code
bash
kubectl get configmap -n <ns> prefect-worker-base-job-template -o yaml
4. If available, flow-run pod logs around startup Relevant docs: - Server Helm guide - Security settings - Work pools and worker infrastructure If you want, paste the 
baseJobTemplate
ConfigMap next — I suspect that’s where the missing auth propagation is.
m
@Marvin There is only one worker connected to the server, and the worker is throwing errors like this:
Copy code
15:57:43.524 | ERROR   | prefect.flow_runs.observer - Container logs from container 'prefect-job' in pod 'soft-ibex-k86ms-d4r45':
15:57:43.525 | ERROR   | prefect.flow_runs.observer - 15:57:09.387 | WARNING | prefect.events.clients - Unable to connect to '<ws://prefect-server.bosaq.svc.cluster.local:4200/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 403. Set PREFECT_DEBUG_MODE=1 to see the full error.
15:57:43.525 | ERROR   | prefect.flow_runs.observer - Traceback (most recent call last):
15:57:43.526 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
15:57:43.527 | ERROR   | prefect.flow_runs.observer -     return fn(*args, **kwargs)
15:57:43.527 | ERROR   | prefect.flow_runs.observer -            ^^^^^^^^^^^^^^^^^^^
15:57:43.528 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/prefect/cli/_types.py", line 156, in sync_fn
15:57:43.528 | ERROR   | prefect.flow_runs.observer -     return asyncio.run(async_fn(*args, **kwargs))
15:57:43.528 | ERROR   | prefect.flow_runs.observer -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:57:43.529 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run
15:57:43.529 | ERROR   | prefect.flow_runs.observer -     return runner.run(main)
15:57:43.530 | ERROR   | prefect.flow_runs.observer -            ^^^^^^^^^^^^^^^^
15:57:43.530 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
15:57:43.531 | ERROR   | prefect.flow_runs.observer -     return self._loop.run_until_complete(task)
15:57:43.531 | ERROR   | prefect.flow_runs.observer -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:57:43.532 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
15:57:43.532 | ERROR   | prefect.flow_runs.observer -     return future.result()
15:57:43.533 | ERROR   | prefect.flow_runs.observer -            ^^^^^^^^^^^^^^^
15:57:43.533 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/prefect/cli/flow_run.py", line 412, in execute
15:57:43.534 | ERROR   | prefect.flow_runs.observer -     await runner.execute_flow_run(id)
15:57:43.534 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 601, in execute_flow_run
15:57:43.535 | ERROR   | prefect.flow_runs.observer -     async with context:
15:57:43.535 | ERROR   | prefect.flow_runs.observer -                ^^^^^^^
15:57:43.536 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 1559, in __aenter__
15:57:43.537 | ERROR   | prefect.flow_runs.observer -     await self._exit_stack.enter_async_context(self._events_client)
15:57:43.537 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/contextlib.py", line 659, in enter_async_context
15:57:43.538 | ERROR   | prefect.flow_runs.observer -     result = await _enter(cm)
15:57:43.538 | ERROR   | prefect.flow_runs.observer -              ^^^^^^^^^^^^^^^^
15:57:43.539 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/prefect/events/clients.py", line 284, in __aenter__
15:57:43.540 | ERROR   | prefect.flow_runs.observer -     await self._reconnect()
15:57:43.540 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/prefect/events/clients.py", line 311, in _reconnect
15:57:43.541 | ERROR   | prefect.flow_runs.observer -     self._websocket = await self._connect.__aenter__()
15:57:43.542 | ERROR   | prefect.flow_runs.observer -                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:57:43.543 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/websockets/asyncio/client.py", line 587, in __aenter__
15:57:43.543 | ERROR   | prefect.flow_runs.observer -     return await self
15:57:43.544 | ERROR   | prefect.flow_runs.observer -            ^^^^^^^^^^
15:57:43.545 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/prefect/_internal/websockets.py", line 135, in _proxy_connect
15:57:43.545 | ERROR   | prefect.flow_runs.observer -     proto = await self.__await_impl__()
15:57:43.546 | ERROR   | prefect.flow_runs.observer -             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:57:43.547 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/websockets/asyncio/client.py", line 543, in __await_impl__
15:57:43.547 | ERROR   | prefect.flow_runs.observer -     await self.connection.handshake(
15:57:43.547 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/websockets/asyncio/client.py", line 114, in handshake
15:57:43.548 | ERROR   | prefect.flow_runs.observer -     raise self.protocol.handshake_exc
15:57:43.549 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/websockets/client.py", line 325, in parse
15:57:43.549 | ERROR   | prefect.flow_runs.observer -     self.process_response(response)
15:57:43.550 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/websockets/client.py", line 142, in process_response
15:57:43.550 | ERROR   | prefect.flow_runs.observer -     raise InvalidStatus(response)
15:57:43.551 | ERROR   | prefect.flow_runs.observer - websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403
m
Your message was too long, here's your message at the allowed limit: There is only one worker connected to the server and the worker is throwing errors like this ```15 57 43 524 | ERROR | prefect flow_runs observer - Container logs from container 'prefect-job' in pod 'soft-ibex-k86ms-d4r45' 15 57 43 525 | ERROR | prefect flow_runs observer - 15 57 09 387 | WARNING | prefect events clients - Unable to connect to '<ws //prefect-server bosaq svc cluster local 4200/api/events/in>' Please check your network settings to ensure websocket connections to the API are allowed Otherwise event data (including task run data) may be lost Reason server rejected WebSocket connection HTTP 403 Set PREFECT_DEBUG_MODE=1 to see the full error 15 57 43 525 | ERROR | prefect flow_runs observer - Traceback (most recent call last) 15 57 43 526 | ERROR | prefect flow_runs observer - File /usr/local/lib/python3 12/site-packages/prefect/cli/_utilities py line 44 in wrapper 15 57 43 527 | ERROR | prefect flow_runs observer - return fn(*args *kwargs) 15 57 43 527 | ERROR | prefect flow_runs observer - ^^^^^^^^^^^^^^^^^^^ 15 57 43 528 | ERROR | prefect flow_runs observer - File /usr/local/lib/python3 12/site-packages/prefect/cli/_types py line 156 in sync_fn 15 57 43 528 | ERROR | prefect flow_runs observer - return asyncio run(async_fn(*args *kwargs)) 15 57 43 528 | ERROR | prefect flow_runs observer - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 15 57 43 529 | ERROR | prefect flow_runs observer - File /usr/local/lib/python3 12/asyncio/runners py line 195 in run 15 57 43 529 | ERROR | prefect flow_runs observer - return runner run(main) 15 57 43 530 | ERROR | prefect flow_runs observer - ^^^^^^^^^^^^^^^^ 15 57 43 530 | ERROR | prefect flow_runs observer - File /usr/local/lib/python3 12/asyncio/runners py line 118 in run 15 57 43 531 | ERROR | prefect flow_runs observer - return self _loop run_until_complete(task) 15 57 43 531 | ERROR | prefect flow_runs observer - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 15 57 43 532 | ERROR | prefect flow_runs observer - File /usr/local/lib/python3 12/asyncio/base_events py line 691 in run_until_complete 15 57 43 532 | ERROR | prefect flow_runs observer - return future result() 15 57 43 533 | ERROR | prefect flow_runs observer - ^^^^^^^^^^^^^^^ 15 57 43 533 | ERROR | prefect flow_runs observer - File /usr/local/lib/python3 12/site-packages/prefect/cli/flow_run py line 412 in execute 15 57 43 534 | ERROR | prefect flow_runs observer - await runner execute_flow_run(id) 15 57 43 534 | ERROR | prefect flow_runs observer - File /usr/local/lib/python3 12/site-packages/prefect/runner/runner py line 601 in execute_flow_run 15 57 43 535 | ERROR | prefect flow_runs observer - async with context 15 57 43 535 | ERROR | prefect flow_runs observer - ^^^^^^^ 15 57 43 536 | ERROR | prefect flow_runs observer - File /usr/local/lib/python3 12/site-packages/prefect/runner/runner py line 1559 in aenter 15 57 43 537 | ERROR | prefect flow_runs observer - await self _exit_stack enter_async_context(self _events_client) 15 57 43 537 | ERROR | prefect flow_runs observer - File /usr/local/lib/python3 12/contextlib py line 659 in enter_async_context 15 57 43 538 | ERROR | prefect flow_runs observer - result = await _enter(cm) 15 57 43
m
@Marvin There is only one worker connected to the server, and the worker is throwing errors like this:
Copy code
15:57:43.524 | ERROR   | prefect.flow_runs.observer - Container logs from container 'prefect-job' in pod 'soft-ibex-k86ms-d4r45':
15:57:43.525 | ERROR   | prefect.flow_runs.observer - 15:57:09.387 | WARNING | prefect.events.clients - Unable to connect to '<ws://prefect-server.bosaq.svc.cluster.local:4200/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 403. Set PREFECT_DEBUG_MODE=1 to see the full error.
15:57:43.525 | ERROR   | prefect.flow_runs.observer - Traceback (most recent call last):
15:57:43.526 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/prefect/cli/_utilities.py", line 44, in wrapper

# Part of stack trace redacted as the message is too long otherwise
...

15:57:43.549 | ERROR   | prefect.flow_runs.observer -     self.process_response(response)
15:57:43.550 | ERROR   | prefect.flow_runs.observer -   File "/usr/local/lib/python3.12/site-packages/websockets/client.py", line 142, in process_response
15:57:43.550 | ERROR   | prefect.flow_runs.observer -     raise InvalidStatus(response)
15:57:43.551 | ERROR   | prefect.flow_runs.observer - websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403
m
✅ thought for 11.1 seconds
That narrows it down a lot. This is not primarily the worker control connection. The failing client is the flow-run container (
prefect-job
) trying to open the events websocket at:
Copy code
text
<ws://prefect-server.bosaq.svc.cluster.local:4200/api/events/in>
So the likely issue is: Your spawned job pods do not have the auth config injected, even if the worker deployment itself does. Because server basic auth is enabled, the run container must also have:
Copy code
text
PREFECT_API_URL=<http://prefect-server.bosaq.svc.cluster.local:4200/api>
PREFECT_API_AUTH_STRING=<same auth-string as server>
Without that, the run process tries to connect to 
/api/events/in
, gets rejected with 403, and you see the server-side websocket warning. Why your current setup can still fail You’re using a custom base job template:
Copy code
yaml
config:
  baseJobTemplate:
    existingConfigMapName: prefect-worker-base-job-template
That is the most likely place where auth propagation got lost. The worker Helm settings can configure the worker pod, but the job pod template used for flow runs may need to explicitly include the API auth env vars. What to inspect now Please check the actual env vars inside one failing flow-run pod:
Copy code
bash
kubectl exec -n bosaq <flow-run-pod-name> -- printenv | grep -E 'PREFECT_API_URL|PREFECT_API_AUTH_STRING|PREFECT_API_KEY'
I expect one of these to be true: - 
PREFECT_API_AUTH_STRING
is missing - 
PREFECT_API_URL
is missing/wrong - 
PREFECT_API_KEY
is present and interfering Also inspect the base job template ConfigMap:
Copy code
bash
kubectl get configmap -n bosaq prefect-worker-base-job-template -o yaml
You want to see the generated pod/job env include the auth settings for the run container. What the fix usually looks like If your custom base job template overrides env, add the required env vars to the job container template, for example:
Copy code
yaml
env:
  - name: PREFECT_API_URL
    value: "<http://prefect-server.bosaq.svc.cluster.local:4200/api>"
  - name: PREFECT_API_AUTH_STRING
    valueFrom:
      secretKeyRef:
        name: prefect-basicauth
        key: auth-string
And ensure there is no 
PREFECT_API_KEY
in that container. One more important note The traceback from inside the run container suggests the run is using a client behavior that still raises on events websocket failure instead of gracefully degrading. There are Prefect issues/PRs around this websocket/auth behavior: - Issue 20775 - Issue 21099 - PR 21269 So there are two layers here: 1. Root cause: flow-run pod likely missing auth env 2. Product behavior: the run should ideally degrade more gracefully Next best step Please paste either: 1. the output of
Copy code
bash
kubectl exec -n bosaq <flow-run-pod-name> -- printenv | grep -E 'PREFECT_API_URL|PREFECT_API_AUTH_STRING|PREFECT_API_KEY'
or 2. the 
prefect-worker-base-job-template
ConfigMap That should let us pinpoint the exact missing field.
m
@Marvin I see the following configured on the flow run jobs, but those are running an older version of prefect (<3.6.13), so that seems to be the issue as described in issue 21099. However, that still does not explain the error log in the prefect-worker
Copy code
spec:
      containers:
        - args:
            - prefect
            - flow-run
            - execute
          env:
            - name: PREFECT_DEBUG_MODE
              value: 'False'
            - name: PREFECT_API_URL
              value: <http://prefect-server.bosaq.svc.cluster.local:4200/api>
            - name: PREFECT_API_AUTH_STRING
              valueFrom:
                secretKeyRef:
                  key: auth-string
                  name: prefect-basicauth
m
✅ thought for 12.1 seconds
Yes — that explains the flow-run job failures. If those job containers are running Prefect < 3.6.14 against a server with 
PREFECT_SERVER_API_AUTH_STRING
enabled, they can hit the exact websocket auth/subprotocol problem described in: - Issue 21099 - Issue 20775 - PR 21269 So for the run containers, the fix is simply: Upgrade the Prefect version in the flow-run image to >= 3.6.14 Ideally, match your server/worker at 
3.7.5
. --- For the prefect-worker log on the server side:
Copy code
text
WebSocket connection rejected: 'prefect' subprotocol required when auth is configured.
that still can be explained by those older flow-run containers. Why: the server log message is generic and emitted whenever any websocket client connects to a protected websocket endpoint without the required 
prefect
subprotocol. It is not exclusive to the worker control channel. So if your older run containers are repeatedly trying to connect to 
/api/events/in
, the server will log that warning, and you may observe it while the worker is also reporting run failures. In other words, the message can be caused by the run pods, not by the worker deployment itself. What this means in practice You likely have: - worker deployment: OK - server deployment: OK - flow-run job images: too old That combination is enough to produce: - worker-observed run errors - server websocket rejection warnings How to confirm Check timestamps: - server warning - worker observer error about 
prefect-job
- flow-run pod startup If they line up, that’s your explanation. You can also verify the worker itself is healthy by confirming it stays connected to its work pool and continues polling/submitting work aside from these failed runs. --- Recommended fix Update the image used by your deployments / work pool job template so the executed flow-run container uses Prefect 
3.7.5
or at least 
>=3.6.14
. If you set the image in your deployment or work pool template, make sure that image actually contains the newer Prefect version. --- One subtle point Issue 21099 specifically describes the old client crash behavior on websocket rejection. Newer behavior degrades more gracefully for events, but if you are on 
<3.6.14
in the run image, you can still get the crash path you’re seeing. --- So short answer: Yes — the old Prefect version in the flow-run image fully explains both: 1. the 
HTTP 403
websocket failure in the run pod 2. the corresponding websocket rejection warning in the server logs If you want, I can help you trace where the old Prefect version is coming from in your deployment/work-pool config and show exactly how to update the run image.
m
@Marvin so you mean that the error log I see in the worker is actually just a mirror of the error in the flow run job?
m
✅ thought for 5.7 seconds
Yes — that’s very likely. The log you showed from the worker:
Copy code
text
ERROR | prefect.flow_runs.observer - Container logs from container 'prefect-job' in pod ...
...
WARNING | prefect.events.clients - Unable to connect to 'ws://.../api/events/in'
...
websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403
means the worker is streaming/forwarding logs from the flow-run pod. It is not necessarily the worker process itself failing to open that websocket. The key clue is this prefix:
Copy code
text
prefect.flow_runs.observer - Container logs from container 'prefect-job' in pod ...
That indicates: - the worker launched the Kubernetes job - the worker is observing that pod - the worker is emitting the pod’s stderr/stdout into its own logs So the error is originating inside the flow-run container, and the worker is just reporting it. In short: - server log = server rejecting old client websocket connection - worker log = worker relaying the failing flow-run pod logs - actual broken client = the old Prefect version inside the flow-run job image If you want, I can help you verify exactly which image/tag your work pool is launching and where to update it.
✅ 1
750 Views
Open in Slack
PreviousNext
Powered by