https://prefect.io
Join Slack
<@ULVA73B9P> is there a managed automation available for cancelling jobs that are pending for &gt; 1...
# ask-marvin
s
@Marvin is there a managed automation available for cancelling jobs that are pending for > 10+ minutes?
m
✅ thought for 92.7 seconds
Not as a managed automation, no. In Prefect Cloud, the built-in managed automation is for unresponsive/zombie runs that are stuck in 
Running
— not for runs/jobs stuck in 
Pending
. If you want to cancel runs that stay 
Pending
for more than 10 minutes, the recommended approach is to create a custom automation with a proactive event trigger. Example pattern: - watch for 
prefect.flow-run.Pending
- expect 
prefect.flow-run.Running
or 
prefect.flow-run.Crashed
- if neither happens within 600 seconds - run 
CancelFlowRun
Example YAML:
Copy code
yaml
name: Cancel long-pending flow runs
enabled: true
trigger:
  type: event
  posture: Proactive
  after:
    - prefect.flow-run.Pending
  expect:
    - prefect.flow-run.Running
    - prefect.flow-run.Crashed
  for_each:
    - prefect.resource.id
  threshold: 1
  within: 600
  match:
    prefect.resource.id:
      - "prefect.flow-run.*"
actions:
  - type: cancel-flow-run
A small terminology note: - 
Pending
= submitted, waiting on execution/infrastructure - 
Late
= scheduled start time passed, but the run has not entered 
Pending
yet So if your concern is specifically “worker never picked this up / infra never started”, 
Pending
is usually the right target. If your concern is “scheduled run missed its start”, that’s more of a 
Late
case. Docs: - Automations - Event triggers - Creating automations - Managed zombie-flow detection example If you want, I can also give you: - a UI-based setup walkthrough, or - a version scoped to work-pool/job variables or only certain deployments.
s
@Marvin can this automation be used to cancel any flow run that has been pending for longer than 10+ minutes?
m
✅ thought for 4.9 seconds
Yes — with one important caveat. That automation pattern can be used to cancel any flow run that remains in 
Pending
for more than 10 minutes, as long as the automation’s match criteria include those runs. If you use:
Copy code
yaml
match:
  prefect.resource.id:
    - "prefect.flow-run.*"
then it will apply broadly to all flow runs in that workspace. A few notes: - It acts on flow runs, not individual infrastructure “jobs” directly. - It cancels the run after it has entered 
Pending
 and failed to transition out within the time window. - It will not catch runs that are stuck in `Scheduled`/`Late` and never make it to 
Pending
. - You can narrow scope by deployment, work pool, tags, etc. if you do not want it workspace-wide. So the short answer is: yes, for any flow run that actually reaches 
Pending
and stays there longer than 10 minutes. Docs: - Event triggers - Automations If you want, I can help you write a version that targets: - all runs in the workspace, - only one deployment, - only one work pool, or - only tagged runs.
s
@Marvin what would be a good name for the custom automation? • Cancel long running pending jobs • Cancel jobs stuck in pending for longer than 10 minutes • Cancel flow run stuck in pending for longer than 10 minutes
m
✅ thought for 8.1 seconds
I’d avoid 
job
in the name unless your team consistently uses that term, since the automation is acting on flow runs. Best of your options: - Cancel flow runs stuck in Pending for longer than 10 minutes Slightly cleaner versions: - Cancel flow runs pending for more than 10 minutes - Cancel stuck Pending flow runs after 10 minutes - Cancel flow runs stuck in Pending >10m My recommendation for clarity: - Cancel flow runs pending for more than 10 minutes Why: - uses Prefect’s actual object name: 
flow runs
- avoids ambiguity with worker/infrastructure jobs - concise but still specific If you want, I can also suggest: - a user-facing name and - a shorter ops/internal naming convention version.
s
@Marvin can a custom automation have more than one action? for example cancelled a flow run and send a notification to a designated slack channel
m
✅ thought for 7.5 seconds
Yes — a custom automation can have multiple actions. So you can absolutely do something like: - cancel the flow run - send a Slack notification That’s a common pattern. A couple of practical notes: - Actions are attached to the same trigger, so when the automation fires, all configured actions are executed. - You’ll want to verify the Slack notification action is configured with your Slack integration/webhook details in your environment. - In practice, it’s usually wise to include enough context in the Slack message to identify the run, deployment, and why it was cancelled. Relevant docs: - Automations - Creating automations A typical setup would be:
Copy code
yaml
actions:
  - type: cancel-flow-run
  - type: send-notification
    # notification configuration here
If you want, I can help draft the full automation for: - cancel + Slack notification, either - in YAML, or - as a step-by-step UI configuration.
s
@Marvin what would you suggest including in the slack message for a flow run that is pending for > 10+ minutes and has been cancelled? The 
flow_run_id
, 
flow_run_name
, 
deployment_name
?
m
✅ thought for 8.0 seconds
Yes — those are good core fields. I’d suggest including: - 
flow_run_name
- 
flow_run_id
- 
deployment_name
- the reason it was cancelled, e.g. “Pending for more than 10 minutes” - a link to the flow run in Prefect Cloud, if available - optionally: - 
flow_name
- 
work_pool_name
- scheduled start time - how long it was pending A good Slack message should help someone answer: 1. What was cancelled? 2. Why was it cancelled? 3. Where do I go inspect it? A practical message shape:
Copy code
text
Prefect automation cancelled a flow run that remained in Pending for more than 10 minutes.

Flow run: <flow_run_name>
Flow run ID: <flow_run_id>
Deployment: <deployment_name>
Reason: Exceeded 10-minute Pending threshold
View run: <run URL>
If you want a slightly more operational version:
Copy code
text
:warning: Prefect cancelled a flow run stuck in Pending.

Flow run: <flow_run_name>
Flow run ID: <flow_run_id>
Flow: <flow_name>
Deployment: <deployment_name>
Work pool: <work_pool_name>
Reason: Pending for >10 minutes
Run URL: <run URL>
My recommendation: keep it short, but always include: - run name - run ID - deployment name - reason - run URL That usually gives responders enough to investigate quickly. If you want, I can help you turn this into a Prefect automation notification template using the actual event/resource variables available in the message body.
s
@Marvin How to managed automations to detect and respond to zombie flow work in Prefect version 2.20.16?
m
✅ thought for 101.2 seconds
In Prefect 2.20.16, there are not managed automations for zombie flow detection. Instead, the way to handle this is to create a custom automation in Prefect Cloud that detects flow runs that enter 
Running
but do not reach a terminal state within some time window. A common pattern is: - `after`: 
prefect.flow-run.running
- `expect`: terminal events like 
prefect.flow-run.completed
and 
prefect.flow-run.cancelled
- `posture`: 
Proactive
- `within`: your timeout window, like 1 hour - action(s): cancel the flow run, send a notification, etc. Conceptually: - start watching when a run enters 
Running
- if no completion/cancellation event happens within the configured window - fire the automation Example shape:
Copy code
python
from datetime import timedelta

from prefect.automations import Automation
from prefect.events.schemas.automations import EventTrigger, Posture
from prefect.server.events.actions import CancelFlowRun

automation = Automation(
    name="Cancel zombie flow runs",
    trigger=EventTrigger(
        match={"prefect.resource.id": "prefect.flow-run.*"},
        for_each={"prefect.resource.id"},
        after={"prefect.flow-run.running"},
        expect={"prefect.flow-run.completed", "prefect.flow-run.cancelled"},
        posture=Posture.Proactive,
        within=timedelta(hours=1),
        threshold=1,
    ),
    actions=[CancelFlowRun()],
).create()
A few notes for 2.20.16: - This is a Prefect Cloud feature. - It’s custom automation, not built-in managed automation. - You can attach multiple actions, like cancel + Slack/email notification. - 
timeout_seconds
on a flow is useful, but it’s different — that handles flow execution timeouts, not orchestration-level “zombie” detection in Cloud. Docs: - Automations concepts - Automations guide - Events and triggers - Prefect Cloud Automations API One important note: if you’re planning new automation work, I’d recommend using Prefect 3.x where possible, since that’s the current default platform direction. If you want, I can help you write the exact 2.20.16 automation for: - zombie 
Running
flows, or - 
Pending
flows older than 10 minutes.
s
@Marvin I see this in the Prefect UI and we are running Prefect v2.20.16
m
✅ thought for 9.4 seconds
Yes — if you see it in the UI on Prefect Cloud while running Prefect 
2.20.16
, that makes sense. The important distinction is: - Managed automations are configured in the UI by Prefect Cloud - they are Cloud-side features - they are not tied strictly to your Python package version in the same way SDK APIs are So even if your flows are running with Prefect 
2.20.16
, you may still see managed automation capabilities in the Cloud UI. For zombie flow detection specifically: - if the UI shows a managed automation for detecting unresponsive/zombie flow runs, then yes, you can use that even though your runtime is on 
2.20.16
- that feature is part of Prefect Cloud’s control plane behavior, not just the local SDK version So the practical answer is: - Yes, trust what you see in the UI - if the managed automation is available in your workspace, you can use it - your 
2.20.16
runtime does not necessarily prevent that If you want, send me the exact label/text you see in the UI, and I can help interpret whether it’s: - the built-in zombie/unresponsive run managed automation, or - a regular custom automation template.
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by