Thanks — that helps narrow it down. So the picture is: - HA server with Redis-backed coordination - Redis lease storage - flow/task runs end up in Cancelled state That combination suggests the cancellations may be coming from orchestration / lease-loss behavior rather than simple worker crashes. A few things would help distinguish the source: 1. Where are you seeing the cancellation happen? - Immediately after submission? - While the run is actively executing? - After some period of idleness / long runtime? 2. Do the server logs show leadership changes or lease renewal failures? In HA mode with Redis leases, I’d look for messages around: - lease acquisition failure - lease renewal failure - leadership transitions - Redis connectivity interruptions / timeouts 3. Do worker logs show the run being cancelled explicitly? Useful clues are log lines indicating: - a cancellation signal was received - infrastructure was told to terminate - polling the API returned a

CANCELLED

state 4. Are these cancellations correlated with Redis instability? Even brief Redis outages or latency spikes can matter if they interfere with lease renewal. 5. Are the cancellations affecting all runs or only long-running ones? If mostly long-running runs, timing/heartbeat/lease-expiry patterns become more likely. What I’d check next: - Prefect server logs around the timestamp of a cancelled run - Worker logs for the same run ID - Redis logs/metrics for disconnects, failovers, or latency spikes - whether multiple API instances are repeatedly flapping leadership If you want, I can help you build a focused debugging checklist for HA + Redis lease storage cancellations, or help interpret specific server/worker log snippets if you paste them here.