<@ULVA73B9P> I want late runs to be cancelled afte...
# ask-marvin
v
@Marvin I want late runs to be cancelled after 5 minutes. In my case, runs tend to accumulate in AwaitingConcurrencySlot state due to a concurreny limit of 1 on each flow. I created an automation to cancel all flow runs that stay in AwaitingConcurrencySlot state for 5 minutes, but it flows are never cancelled and stay in AwaitingConcurrencySlot forever. Why is this happening ?
m
thought for 306.8 seconds
Short answer: your automation’s condition never becomes true. Why: When a run is waiting on a concurrency limit, Prefect puts it into the AwaitingConcurrencySlot state and the scheduler periodically re-evaluates it. Each re-evaluation re-applies the same state with a fresh timestamp. Automations that use “state maintained for 5 minutes” rely on the state not changing at all; because the backend keeps re-setting AwaitingConcurrencySlot, the “maintained for” timer keeps getting reset and never reaches 5 minutes. That’s why your runs sit there “forever” and the cancellation never fires. What to do instead: - Use a proactive “did not happen within” automation: cancel the run if it does not enter Running within 5 minutes of being scheduled/created. This trigger is based on the absence of the Running event, so it’s not affected by the state re-applies while waiting for a concurrency slot. - In the UI, create an automation with: - Trigger: “Flow run entered Running” did not occur within 5 minutes after “Flow run was scheduled” (or “created”, depending on your setup) - Action: “Change state to Cancelled” - Note: The “Late” state in Prefect is a specific state for scheduled runs that miss their start; runs waiting in AwaitingConcurrencySlot generally will not become Late, so targeting Late won’t help here. Helpful docs: - Event triggers and proactive automations: Event triggers (Automations) - Concurrency limits overview: Global concurrency limits If you’d like, I can outline the exact clicks in the UI to set up that proactive automation or provide an API snippet to create it programmatically.
v
@Marvin yes please
m
thought for 128.4 seconds
Great — here are the exact UI steps to set up a proactive automation that cancels runs that don’t start within 5 minutes (it will catch runs stuck in AwaitingConcurrencySlot): UI steps (Prefect 3.x) 1) Go to Automations in the Prefect UI and click New automation. 2) Name it something like “Cancel runs that don’t start within 5m”. 3) Trigger - Choose Proactive. - Select Event did not happen within. - Anchor event (happened): pick one of: - Flow run was scheduled (best for deployment-scheduled runs), or - Flow run was created (use this if you also trigger runs ad-hoc and they may not emit Scheduled). - Missing event (did not happen): Flow run entered Running. - Within: 5 minutes. 4) Scope / Filters - Resource: Flow run. - Narrow the scope so you only target the runs you care about: - By Deployment: choose your specific deployment(s), or - By Flow: choose your flow, or - By Tag(s): if you tag these runs, filter on those tags. - This ensures you don’t cancel unrelated runs. 5) Action - Choose Change flow run state (or Cancel flow run). - Set the target state to Cancelled. - Optional: add a message like “Cancelled by automation after waiting >5m to start”. 6) Save and Enable the automation. Notes and tips - Why this works: It keys off the absence of the “Running” event within 5 minutes, so periodic re-application of AwaitingConcurrencySlot won’t reset the timer. - Which anchor event to pick: - If your runs are created by a deployment schedule, use “Flow run was scheduled”. - If you also kick off runs manually or via API without a schedule, use “Flow run was created”. - Scope carefully so you don’t cancel long warm-up flows that are slow to start for legitimate reasons. - Test: Saturate the concurrency (limit=1), start another run, confirm it cancels ~5 minutes after the selected anchor event. Docs for reference - Event triggers (Automations): Event triggers (Automations) - Concurrency limits: Global concurrency limits If you prefer to create this programmatically via the API, I can draft a JSON payload for the Automations endpoint based on your exact scope (deployment/flow/tags). Just let me know which you want to filter by.