It’s an auth mismatch on the events WebSocket. In Prefect 3 self‑hosted, the events stream at

/api/events/in

uses a shared auth string when the server is configured with it. Your Kubernetes pod (operator/worker) must send the same value to authenticate. What to do (assuming Prefect 3.x): 1) Check the server’s auth setting in your docker-compose - On the container that runs the API (and services), verify whether

PREFECT_SERVER_API_AUTH_STRING

is set and note its exact value. - If it is set, every external client must use the same value. If you don’t want auth, remove it from the server (and restart), but using auth is recommended. 2) Configure the Kubernetes pod with the matching client env vars - Set

PREFECT_API_URL

to your API base URL (including

/api

). - Set

PREFECT_API_AUTH_STRING

to exactly the same value as

PREFECT_SERVER_API_AUTH_STRING

. Example (with a Secret + env refs):

apiVersion: v1
kind: Secret
metadata:
  name: prefect-credentials
type: Opaque
stringData:
  PREFECT_API_URL: <http://myprivatedomain:4200/api>
  PREFECT_API_AUTH_STRING: "admin:mypassword"   # must match the server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-operator-or-worker
spec:
  template:
    spec:
      containers:
        - name: prefect
          image: prefecthq/prefect:3-latest
          env:
            - name: PREFECT_API_URL
              valueFrom:
                secretKeyRef:
                  name: prefect-credentials
                  key: PREFECT_API_URL
            - name: PREFECT_API_AUTH_STRING
              valueFrom:
                secretKeyRef:
                  name: prefect-credentials
                  key: PREFECT_API_AUTH_STRING
          # command/args as appropriate for your operator or worker

Notes and checks: - The error shows

<ws://myprivatedomain:4200/api/events/in>

, which is derived from

PREFECT_API_URL

. That’s good. If your endpoint is HTTPS, set

PREFECT_API_URL=<https://myprivatedomain:4200/api>

so it upgrades to

wss://.../events/in

. - Make sure port 4200 is reachable from the cluster. - Do not set

PREFECT_API_KEY

when self‑hosting; that’s for Prefect Cloud and won’t work here. - If you want more detail while testing, set

PREFECT_DEBUG_MODE=1

in the pod to see the full auth handshake messages. If you’re using the Prefect Kubernetes Operator (the

kopf

log suggests you are), add these env vars to the operator’s Deployment (or via your Helm values) so the operator can authenticate to your self‑hosted API. If you’re running a Kubernetes Worker instead, add the same env vars to that worker’s Deployment. If you can share: - whether this pod is the Prefect Kubernetes Operator or a Worker - the

PREFECT_SERVER_API_AUTH_STRING

value configured on your docker-compose server (you can redact it, just confirm whether it’s set) - the current

PREFECT_API_URL

and

PREFECT_API_AUTH_STRING

in the k8s pod …I can tailor the exact patch/Helm values you need.

Yes. For the Prefect Kubernetes Operator/Observer, you can disable its events connection entirely. Key options: - Stop the observer (prevents any events client from being created) - Set this on the operator/worker pod: -

PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false

- Keep the observer running but stop pod event replication - This still initializes the events client, but won’t emit pod/job events: -

PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_REPLICATE_POD_EVENTS=false

If your goal is to avoid the WebSocket connection (and the auth error), use the first option: set

PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false

on that pod and restart it. That prevents the operator from creating the events client at startup, so it won’t try to connect to

/api/events/in

. If you’d like, share whether this pod is the operator or a worker, and I can provide a snippet (Deployment/Helm values) showing exactly where to add the env var.

You’re seeing that because the Kubernetes observer is separate from Prefect’s global EventsWorker. Disabling the observer stops K8s pod event replication, but the EventsWorker still starts inside the worker process and tries to open the

/api/events/in

websocket whenever

PREFECT_API_URL

is set. What you can do: Option A (recommended if you just want the noise gone) - Supply the correct auth for events so the connection succeeds: - On your worker/operator pod, set

PREFECT_API_AUTH_STRING

to exactly match the server’s

PREFECT_SERVER_API_AUTH_STRING

. - This keeps events enabled but stops the warnings/errors. Option B (server-side, no auth) - Remove

PREFECT_SERVER_API_AUTH_STRING

from your self-hosted server/services so the WS does not require auth. Clients will connect without errors. Only do this if that’s acceptable for your environment. Option C (truly disable events from this pod) - There is no single env var to disable the EventsWorker today; it will run as long as

PREFECT_API_URL

is set (and you need that for the worker to function). - If you must disable it, use a small customization to force a NullEventsClient before the CLI starts. One robust approach is a sitecustomize shim in your worker image: 1) Build a tiny layer that adds sitecustomize.py:

# sitecustomize.py
  try:
      from prefect.events.worker import EventsWorker
      from prefect.events.clients import NullEventsClient
      EventsWorker.set_client_override(NullEventsClient)
  except Exception:
      pass

2) Put this file on PYTHONPATH in your worker image (Python auto-imports sitecustomize on startup). Then run the normal

prefect worker start ...

command. This will suppress all event emission (no websocket attempts). Why your current change didn’t help -

PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false

only disables the K8s observer. The EventsWorker is independent and still tries to send events because your pod has

PREFECT_API_URL

set. Source refs - Events client/WebSocket auth handshake: https://github.com/PrefectHQ/prefect/blob/main/src/prefect/events/clients.py - EventsWorker decision logic: https://github.com/PrefectHQ/prefect/blob/main/src/prefect/events/worker.py - K8s observer init (shows it’s separate from EventsWorker): https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-kubernetes/prefect_kubernetes/observer.py - K8s observer settings (enabled/replicate_pod_events): https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-kubernetes/prefect_kubernetes/settings.py If you want, I can share a quick patch to your Deployment/Helm values to add

PREFECT_API_AUTH_STRING

, or a minimal Dockerfile snippet for the sitecustomize approach.

Thanks for the details — the fact that you set both

PREFECT_API_AUTH_STRING

(client) and

PREFECT_SERVER_API_AUTH_STRING

(server) to the same value is exactly right. The “auth_token” wording in the log is generic; self‑hosted still requires that same shared token when the server is configured with it. Given you’re still seeing “Unable to authenticate,” the most common causes are: 1) The pod env isn’t what we think - Verify the worker pod actually has the values you expect (no typos/whitespace):

kubectl -n um-demo exec -it <pod-name> -- sh -lc 'env | sort | grep "^PREFECT_"'

- Double‑check `PREFECT_API_URL`: - Must include the

/api

suffix - Must use the correct scheme/port for how you expose the API - If you terminate TLS at a proxy and access via https externally, set

PREFECT_API_URL=<https://another_domain:4300/api>

(this makes the client use wss://). - If you truly use plain http,

http://...

is fine. 2) Reverse proxy/ingress is stripping the WebSocket subprotocol - When

PREFECT_SERVER_API_AUTH_STRING

is set, the server requires the WebSocket subprotocol

prefect

. If a proxy drops

Sec-WebSocket-Protocol: prefect

, the server will close the connection before the auth handshake completes — the client will log an auth failure and “no close frame”. - If you’re using NGINX or another L7 in front of Prefect, ensure it forwards WebSocket headers and the subprotocol. For NGINX, something like:

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;

- For NGINX Ingress Controller, ensure it’s up to date (it handles ws upgrades automatically) and, if you use custom headers via annotations, preserve

Sec-WebSocket-Protocol

. 3) Typo/whitespace in the token on either side - The token comparison is exact. If you’re storing it in a Secret, confirm no stray whitespace:

kubectl -n <ns> get secret <secret> -o jsonpath='{.data.PREFECT_API_AUTH_STRING}' | base64 -d | sed -n l

You should see the value with line-end marker

$

and no extra spaces or quotes. 4) The API URL is not actually the same endpoint your server is bound to - Your logs show

<ws://another_domain:4300/api/events/in>

. Make sure the server is reachable there and that the same container running the API has

PREFECT_SERVER_API_AUTH_STRING

set to the value you expect. If you split “API” and “services” into separate containers, the auth string must be present in the API container specifically. 5) TLS mismatch - If you access via HTTPS externally, use

https://...

in

PREFECT_API_URL

(so the client uses wss://). If you use a self‑signed cert, set:

PREFECT_API_TLS_INSECURE_SKIP_VERIFY=true

Quick ways to pinpoint the issue A) Turn on Prefect debug logs in the pod

PREFECT_DEBUG_MODE=1

This will print the handshake step and typically shows whether the server closed due to a missing subprotocol vs an invalid token. B) Test the WebSocket by hand (bypassing your worker) - From a machine that can reach the API, use wscat/websocat and include the subprotocol:

wscat -c <ws://another_domain:4300/api/events/in> -s prefect
# then paste:
{"type":"auth","token":"<exact PREFECT_SERVER_API_AUTH_STRING value>"}

You should receive:

{
  "type": "auth_success"
}

If this succeeds directly but your worker still fails, the problem is your proxy/ingress path from the cluster to the API (likely the subprotocol header being dropped). If this fails the same way, the problem is server-side config or the token value itself. C) Check server logs at the time of connection attempts - The Prefect API process will log if it rejects the socket due to protocol/auth; that can confirm whether it saw the subprotocol and what it did next. If you don’t actually care about events from this pod and just want to suppress the warnings, two safe alternatives:

- Remove

PREFECT_SERVER_API_AUTH_STRING

from the API container (disables WS auth requirement). Clients will connect without auth. - Force-disable event emission in the worker image using a tiny sitecustomize shim:

# sitecustomize.py on PYTHONPATH in the worker image
from prefect.events.worker import EventsWorker
from prefect.events.clients import NullEventsClient
EventsWorker.set_client_override(NullEventsClient)

If you can share: - Your current

PREFECT_API_URL

value - Whether you’re going through an ingress/proxy (and which one) - The env dump from the worker pod (

env | grep PREFECT_

) …I can spot the issue and provide an exact ingress snippet or config patch. Useful references: - Server WS auth and subprotocol requirement: https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/utilities/subscriptions.py - Client auth handshake: https://github.com/PrefectHQ/prefect/blob/main/src/prefect/events/clients.py - Events decision logic in worker: https://github.com/PrefectHQ/prefect/blob/main/src/prefect/events/worker.py