What you’re seeing is likely expected in this setup. What the warning means - The warning is emitted by Prefect’s Redis stream consumer logic in

prefect-redis

, not by Redis Sentinel directly. - It indicates a transient Redis connection failure while Prefect is listening for events, then it retries automatically. - The relevant code path includes retry + reconnect behavior, so the message itself usually means “temporary disconnect detected, reconnecting”, not “hard failure”. Important clarification - In Prefect 3.x, worker cancellation handling is driven by the event system. - The worker-side cancellation observer itself uses the Prefect API event stream, while server-side event services rely on Redis messaging/streams. - So this warning is generally associated with the event/messaging layer around cancellation, not necessarily a failure of the worker itself. In an HAProxy + Redis Sentinel setup, this can happen during: - Sentinel failover - HAProxy backend switch - idle TCP connection closure - short network interruptions - stale pooled Redis connections If the consumer reconnects and cancellation still works, this is generally normal transient behavior. What to check 1. Do you also see successful reconnect logs? Look for messages like: -

reconnected to Redis successfully

- resumed event processing after the warning 2. How often does it happen? - Occasional: usually OK, especially during failover or infra churn - Frequent/constant: suggests timeout, TCP keepalive, or HAProxy health-check issues 3. Are cancellations actually delayed or missed? - If cancellations still complete correctly, this is mostly noise from reconnect behavior - If cancellations are missed, then it needs deeper investigation Likely mitigations - Ensure HAProxy is configured for long-lived TCP connections to Redis - Avoid aggressive idle timeouts on HAProxy - Enable TCP keepalive - Make sure backend health checks switch cleanly to the current Redis master - Tune Prefect Redis health check interval if needed Example env vars to review

bash
PREFECT_REDIS_MESSAGING_URL=redis://<haproxy-host>:6379/0
PREFECT_REDIS_MESSAGING_HEALTH_CHECK_INTERVAL=10

If you use TLS:

bash
PREFECT_REDIS_MESSAGING_URL=rediss://<haproxy-host>:6379/0

HAProxy considerations For Redis proxying, common best practices are: -

mode tcp

-

option tcpka

- non-aggressive

timeout client

/

timeout server

- health checks that correctly identify the writable/master Redis target Bottom line - Yes, this warning can be expected in a HA Redis setup fronted by HAProxy. - By itself, it does not indicate Prefect is broken. - It becomes concerning only if: - it happens continuously, - reconnect never succeeds, - or flow-run cancellations stop working. Useful references - Prefect deployments/workers docs: Deployments and work pools - Prefect self-hosting docs: Self-host Prefect - Prefect repo: Prefect GitHub If you want, I can help you with a more specific HAProxy config review for Redis/Sentinel, or help determine whether this is harmless in your logs versus an actual cancellation reliability issue.

(though there are several pieces to that)