<@ULVA73B9P> I have noticed, that Prefect process ...
# ask-marvin
r
@Marvin I have noticed, that Prefect process worker is using websockets. Explain reason for websockets.
m
thought for 131.9 seconds
Great observation! In Prefect 3.x, workers (including the process worker) use WebSockets primarily to get push-style, low-latency signals from the API for time-sensitive events. Here’s the why and what: Why WebSockets instead of simple HTTP polling - Push, not pull: WebSockets let the server push events to workers immediately instead of workers polling every N seconds. That reduces latency and API load. - Reliability with ack: The channels Prefect uses support acknowledgments and backfill where appropriate. - Graceful fallback: If a WebSocket can’t be established (e.g., a proxy blocks upgrades), Prefect falls back to periodic HTTP polling so you don’t lose functionality—just the immediacy. What the process worker actually uses WebSockets for - Fast flow-run cancellation signals (main reason) - The worker runs a
FlowRunCancellingObserver
that subscribes over a WebSocket to “flow run cancelling” events. When a run is cancelled while still pending, the worker can tear down infrastructure immediately instead of waiting for a polling interval. - If the WS drops, it automatically switches to polling every
PREFECT_WORKER_CANCELLATION_POLL_SECONDS
. - Source: _observers.py and workers/base.py (worker wiring) - Event and log streaming (consumed by clients/UI) - Prefect exposes WebSockets for real-time events and logs; clients like the UI or
prefect logs --follow
subscribe to these streams. - Endpoints: events stream server/api/events.py, logs stream server/api/logs.py. - Note: the worker itself isn’t the primary consumer of the logs stream; it’s there so observers/clients can follow output live. What the process worker does not use WebSockets for - Work fetching: Flow-run workers (process, docker, k8s) still acquire scheduled flow runs via HTTP (pull). WebSockets are not used to assign flow runs to these workers. Related: Task worker uses WebSockets for work assignment - The task worker subscribes over a WebSocket to receive scheduled task runs for specific task keys with ack semantics. - Source: task_worker.py, server/api/task_runs.py Operational note - If you’re self-hosting behind a reverse proxy, make sure WebSocket upgrades are allowed (e.g.,
Upgrade
and
Connection: upgrade
headers). Otherwise, workers will fall back to polling and you may see slower cancellations. Config knobs to be aware of -
PREFECT_WORKER_ENABLE_CANCELLATION
– enables worker-side cancellation of pending runs (the observer will use WS with a polling fallback). -
PREFECT_WORKER_CANCELLATION_POLL_SECONDS
– fallback interval if the WS cannot be used. - Source: settings/models/worker.py In short: the process worker’s WebSocket is there to receive push-based cancellation signals (and participate in the platform’s real-time event/log streaming), giving you sub-second responsiveness and reduced API churn compared to pure HTTP polling.
r
@Marvin Does it has any consequencies in setup, where Prefect server is in active-active Openshift cluster and process worker is used ?
m
thought for 151.1 seconds
Short answer: you’re fine. Running Prefect Server in an active-active OpenShift cluster with a process worker works well. The only practical consequence of WebSockets is that you must allow WS upgrades through your Route/Ingress. If WebSockets are blocked, the process worker just falls back to polling—so the only impact is slower cancellation of pending runs. What to know for active-active OpenShift - No sticky sessions required - WebSocket subscriptions are stateless at the LB level and Prefect uses Redis-backed messaging; clients can reconnect to any replica. - Ensure WebSocket upgrades are allowed end-to-end - Proxies/Routes must pass
Upgrade
and
Connection: upgrade
headers and not prematurely close idle connections. - Set sane timeouts on OpenShift Routes (to avoid idle WS disconnects) - Example:
Copy code
metadata:
      annotations:
        <http://haproxy.router.openshift.io/timeout|haproxy.router.openshift.io/timeout>: 300s
- Health checks - Point readiness/liveness at
/api/health
. - HA Server layout - Run multiple API replicas with services disabled:
Copy code
prefect server start --host 0.0.0.0 --no-services
- Run exactly one background services instance separately:
Copy code
prefect server services start
- This avoids duplicating schedulers/automations across replicas. Impact on the process worker - Uses WS primarily for fast cancellation of pending flow runs; work fetching is still HTTP. - If WS is available: cancellations are near-instant. - If WS is not available: it automatically switches to polling, so cancellations may take up to
PREFECT_WORKER_CANCELLATION_POLL_SECONDS
. - Logs/events WS endpoints are primarily for UI/clients and do not affect the worker doing its job. Useful config knobs -
PREFECT_WORKER_ENABLE_CANCELLATION
(enable worker-side cancellation) -
PREFECT_WORKER_CANCELLATION_POLL_SECONDS
(fallback polling interval) - Source: settings/models/worker.py References (implementation details) - Process worker cancellation observer wiring: workers/base.py and _observers.py - Events WS endpoint: server/api/events.py - Logs WS endpoint: server/api/logs.py - Self-hosted/HA deployment guidance (incl. WS proxy headers): Self-hosted (docs source) If you want, share your Route/Ingress config and I can sanity-check the WS/timeout bits for OpenShift.