Thanks for sharing the log. Here’s what it means and what you can do about it. What is CancellationCleanup? - It’s a background server service that periodically: - Cancels active task runs that belong to cancelled flow runs - Cancels active subflows when their parent flow is cancelled - It runs on a loop (default every 20s). If one iteration takes longer than the loop interval, you’ll see the warning you posted. - Source: CancellationCleanup service - Loop warning code: LoopService warning Why you’re seeing this - One iteration took ~24s, which is longer than the 20s interval. This is usually benign if it’s occasional. - Common causes: - A burst of recently-cancelled flow runs (the service has more work to do) - Database slowness or load (large flow_run/task_run tables, locks, resource limits) - Network latency to the DB - Very large flows with many child task runs/subflows Mitigations and options - Increase the loop interval (quick, low-risk):

export PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_LOOP_SECONDS=60

Trade-off: cancellations cascade a bit more slowly. - If you don’t rely on cancellation propagation, you can disable it:

export PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false

- Check DB performance: - Ensure your Postgres has adequate CPU/IO and that autovacuum is running effectively - Look for slow queries/locks around flow_run/task_run tables - Advanced: consider additional indexes if you have very large tables and persistent slowness (coordinate with your DBA):

CREATE INDEX IF NOT EXISTS idx_flow_run_state_type_end_time 
      ON flow_run(state_type, end_time);
    CREATE INDEX IF NOT EXISTS idx_flow_run_parent_task_run 
      ON flow_run(parent_task_run_id, state_type);
    CREATE INDEX IF NOT EXISTS idx_task_run_flow_run_state 
      ON task_run(flow_run_id, state_type);

- Operational patterns: - Avoid mass cancellation storms when possible - Reduce extreme fan-out of subflows/tasks if you’re seeing sustained issues Is it a problem? - Occasional warnings are typically fine and the service will catch up. - If the warning is frequent/continuous, it’s a signal to either increase the loop interval or look into database sizing/health. References - Service source: CancellationCleanup - LoopService base and warning: LoopService warning code - Settings reference: - CancellationCleanup docs - LoopService docs - Server services settings Quick questions to help us tailor advice: - What database backend are you using (Postgres version and size), and where is it hosted relative to the API? - Roughly how many flow runs are you cancelling or completing per hour? - Is this warning occasional or persistent?