https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> Suggest some best practice system ope...
# ask-marvin
r
@Marvin Suggest some best practice system operator or administrator type automations. Already have for failed, cancelled, timedout, and crashed flows. As well as for long running flows. And for paused workpools and queues. But what else makes sense to add? Do it for Prefect 3.7.2 and using an automations.yaml. Presumably it can monitor for high concurrency and high waiting/pending? What else?
m
✅ thought for 153.0 seconds
Yes — high concurrency pressure and backlog / waiting / pending are good candidates, and there are a few other operator-focused automations that usually pay off. For Prefect 3.7.2 with 
automations.yaml
, I’d think in a few buckets: 1. Scheduler / orchestration health 2. Worker / work pool / queue health 3. Backlog / congestion signals 4. Deployment readiness 5. Infra-specific failures 6. Noise control / escalation patterns A few good additions beyond what you already have: --- ## Strong candidates to add ### 1. Deployment not ready This is one of the most useful admin signals. If a deployment becomes 
not-ready
, it often means: - bad work pool linkage - no workers available - schedule issues - other orchestration problems Good automation: - notify - optionally pause the deployment so it stops creating more trouble --- ### 2. Worker stopped / disappeared You mentioned paused work pools and queues, but an even earlier signal is when workers themselves stop. This helps catch: - dead containers/pods - hosts restarted - credentials/config drift - autoscaling gone wrong --- ### 3. Work pool not ready Different from paused. A pool can be unpaused but still not actually able to execute work. This is a very good operator/admin alert. --- ### 4. Flows stuck in Pending too long Yes, this is worth monitoring. This often means: - no worker picked up work - queue backlog - infra provisioning delay - work pool misconfiguration - image pull / startup problems You generally want a proactive trigger: “after entering Pending, did not progress within X minutes”. --- ### 5. Flows awaiting concurrency slots Also yes — this is one of the better “system pressure” signals. This is useful for: - global concurrency limits - tag-based concurrency limits - external system throttling - too-small limits after growth If lots of runs are sitting in 
AwaitingConcurrencySlot
, you’ve got congestion. --- ### 6. High lateness for scheduled runs This is a great operator metric because it catches “the system is working, but not on time.” If schedules are firing but runs consistently start late, that often points to: - insufficient workers - queue congestion - infra cold start delays - concurrency bottlenecks In Cloud, metric automations are a strong fit for this. --- ### 7. Low success rate on critical flows/deployments Instead of only alerting on individual failures, add a higher-level “error budget” style automation. Examples: - success rate below 95% over 30 minutes - duration suddenly much higher than normal - lateness above threshold for a sustained window This is better for ops than one-alert-per-failure. --- ### 8. Infrastructure failure events If you’re on Kubernetes / Docker / ECS, listen for infra events too. Examples: - pod evicted - pod crash loop / failed - container exited unexpectedly These are excellent because they explain why Prefect runs are stalling or crashing. --- ### 9. Repeated failures for the same deployment You already alert on failed flows, but a more useful admin automation is: - “N failures for the same deployment in M minutes” That turns user-level failures into an operator signal. Possible action: - notify - optionally pause that deployment - optionally trigger a remediation deployment/flow --- ### 10. Auto-recovery automations Not just alerting — some automations can safely heal. Examples: - resume deployment when it becomes ready again - resume work pool when it becomes ready again - trigger a diagnostic/remediation flow when a deployment repeatedly fails Use these carefully, but they can reduce pager noise. --- ## What I would prioritize first If you want a short practical list, I’d add these next: 1. Deployment not ready 2. Work pool not ready 3. Worker stopped 4. Pending too long 5. Awaiting concurrency slot backlog 6. High lateness / low success-rate metrics for critical flows 7. Repeated deployment failures
8. Infra failure events if using k8s/docker/ecs --- ## Notes on “high concurrency” and “high waiting/pending” ### High concurrency Yes, but usually the better signal is not “concurrency is high” by itself. Better signals are: - many runs entering 
AwaitingConcurrencySlot
- sustained lateness - growing Pending backlog - repeated acquisition pressure on a specific concurrency limit Those reflect user impact more directly. ### High waiting / pending Yes — especially: - 
Pending
too long - 
AwaitingConcurrencySlot
- 
Late
/ lateness metrics - scheduled work not transitioning to Running These are some of the best ops automations because they catch system degradation before users only see failures. --- ## Example 
automations.yaml
Here’s a compact set of admin/operator examples for Prefect 3.7.2. ```yaml automations: - name: Alert on deployment not ready description: Notify when a deployment becomes not ready enabled: true trigger: type: event posture: Reactive expect: - prefect.deployment.not-ready threshold: 1 actions: - type: send-notification block_document_id: YOUR_NOTIFICATION_BLOCK_ID subject: "Deployment not ready: {{ deployment.name }}" body: | Deployment {{ deployment.name }} is not ready. Flow: {{ flow.name }} - name: Alert on work pool not ready description: Notify when a work pool has no effective capacity / readiness enabled: true trigger: type: event posture: Reactive expect: - prefect.work-pool.not-ready threshold: 1 actions: - type: send-notification block_document_id: YOUR_NOTIFICATION_BLOCK_ID subject: "Work pool not ready: {{ work_pool.name }}" body: | Work pool {{ work_pool.name }} is not ready. Check worker health and pool configuration. - name: Alert on worker stopped description: Notify when a worker stops enabled: true trigger: type: event posture: Reactive expect: - prefect.worker.stopped threshold: 1 actions: - type: send-notification block_document_id: YOUR_NOTIFICATION_BLOCK_ID subject: "Worker stopped: {{ event.resource.name }}" body: | Worker {{ event.resource.name }} has stopped. - name: Alert on flow pending too long description: Notify when a flow run does not progress after entering Pending enabled: true trigger: type: event posture: Proactive after: - prefect.flow-run.Pending expect: - prefect.flow-run.Running - prefect.flow-run.Completed - prefect.flow-run.Failed - prefect.flow-run.Crashed threshold: 1 within: 300 for_each: - prefect.resource.id actions: - type: send-notification block_document_id: YOUR_NOTIFICATION_BLOCK_ID subject: "Flow stuck pending: {{ flow_run.name }}" body: | Flow run {{ flow_run.name }} has remained pending for more than 5 minutes. - name: Alert on concurrency backlog description: Notify when flows are piling up awaiting concurrency enabled: true trigger: type: event posture: Proactive expect: - prefect.flow-run.AwaitingConcurrencySlot threshold: 5 within: 60 for_each: - prefect.resource.id actions: - type: send-notification block_document_id: YOUR_NOTIFICATION_BLOCK_ID subject: "Concurrency backlog detected" body: | Multiple flow runs are awaiting concurrency slots. - name: Alert on repeated deployment failures description: Notify when the same deployment fails repeatedly enabled: true trigger: type: event posture: Reactive expect: - prefect.flow-run.Failed threshold: 3 within: 900 for_each: - relateddeploymentprefect.resource.id
actions: - type: send-notification block_document_id: YOUR_NOTIFICATION_BLOCK_ID subject: "Repeated deployment failures" body: | A deployment has had 3 failed flow runs within 15 minutes.
Copy code
---

## Metric automations: very useful if you’re in Cloud

If you’re using Prefect Cloud, I would also add metric-based automations for:

- *lateness* above threshold
- *successes* below threshold
- possibly *duration* above threshold for key deployments

Example pattern:

```yaml
automations:
  - name: Alert on high lateness
    enabled: true
    trigger:
      type: metric
      metric:
        name: lateness
        operator: ">"
        threshold: 600
        range: 3600
        firing_for: 600
    actions:
      - type: send-notification
        block_document_id: YOUR_NOTIFICATION_BLOCK_ID
        subject: "High flow lateness detected"
        body: |
          Scheduled runs are starting more than 10 minutes late over a sustained period.
Copy code
yaml
automations:
  - name: Alert on low success rate for critical flow
    enabled: true
    trigger:
      type: metric
      metric:
        name: successes
        operator: "<"
        threshold: 0.95
        range: 1800
        firing_for: 600
    match:
      prefect.resource.id:
        - "prefect.flow.YOUR_FLOW_ID"
    actions:
      - type: send-notification
        block_document_id: YOUR_NOTIFICATION_BLOCK_ID
        subject: "Low success rate detected"
        body: |
          Success rate dropped below 95% for a critical flow.
--- ## Best-practice guidance so these stay useful A few practical tips: - Prefer aggregate/operator alerts over one-alert-per-run Repeated failures, backlog, lateness, and readiness are usually more actionable. - Scope automations to critical deployments/work pools first Otherwise you can create a lot of noise. - Use different severities For example: - warning: pending > 5 min - critical: deployment not ready - critical: worker stopped in prod pool - warning: concurrency backlog - Use proactive triggers for “stuck” behavior They’re ideal for Pending / waiting cases. - Use 
for_each
carefully It helps you avoid one giant aggregate alert and instead isolate to a deployment / resource. - Avoid over-alerting on transient states A little Pending is normal. Sustained Pending is the problem. --- ## One important caveat I’d be slightly careful with “high pending” as a blanket automation. A short Pending period can be normal depending on your infra. Usually better: - 
Pending > 5m
for normal pools - 
Pending > 10–15m
for slow cold-start environments - use lateness metrics for scheduled workloads --- ## My recommended baseline set If I were building an operator pack for Prefect 3.7.2, I’d include: - deployment not ready - work pool not ready - worker stopped - pending too long - awaiting concurrency slot backlog - repeated failures per deployment - high lateness for critical scheduled deployments - low success rate for critical flows - infra-specific events if on k8s/docker/ecs --- If you want, I can turn this into a full production-ready 
automations.yaml
starter pack with: - Slack notifications - warning vs critical naming - comments - placeholders for IDs - a few optional Cloud-only metric sections clearly separated.
r
@Marvin The multiple awaiting concurrency automation is erroneous triggering, i.e. there are no flow runs awaiting - name: admin-concurrency-backlog description: Notify when flows are piling up awaiting concurrency enabled: true trigger: type: event posture: Proactive expect: - prefect.flow-run.AwaitingConcurrencySlot threshold: 5 within: 60 for_each: - prefect.resource.id actions: - type: send-notification block_document_id: PLACEHOLDER subject: "🚨Concurrency backlog detected" body: | Multiple flow runs are awaiting concurrency slots.
m
✅ thought for 194.1 seconds
Yep — that config is using the wrong trigger semantics for what you want. ## Why it misfires You currently have:
Copy code
yaml
trigger:
  type: event
  posture: Proactive
  expect:
    - prefect.flow-run.AwaitingConcurrencySlot
  threshold: 5
  within: 60
  for_each:
    - prefect.resource.id
The key issue is 
posture: Proactive
. In Prefect automations: - Reactive means “fire when the expected event does happen enough times” - Proactive means “fire when the expected event does not happen enough times within the window” So your automation is effectively closer to: “For each bucket, alert when fewer than 5 
AwaitingConcurrencySlot
events occur within 60s.” That is not “there are currently 5 runs awaiting concurrency”. --- ## Second issue: 
for_each: prefect.resource.id
This groups by flow run id. So the trigger is evaluating per individual flow run, not as a shared backlog counter. That means it is not measuring: - “5 different flow runs are awaiting concurrency” It is more like: - “for this one flow run id, how many matching events happened in the window?” That’s not the right shape for a system backlog alert. --- ## What to use instead ### Best fix: make it Reactive and remove 
for_each
If your goal is: “Alert when 5 flow runs enter 
AwaitingConcurrencySlot
within 60 seconds” use:
Copy code
yaml
- name: admin-concurrency-backlog
  description: Notify when flows are piling up awaiting concurrency
  enabled: true
  trigger:
    type: event
    posture: Reactive
    expect:
      - prefect.flow-run.AwaitingConcurrencySlot
    threshold: 5
    within: 60
  actions:
    - type: send-notification
      block_document_id: PLACEHOLDER
      subject: ":rotating_light: Concurrency backlog detected"
      body: |
        5 or more flow runs entered AwaitingConcurrencySlot within 60 seconds.
This is the closest valid event-based approximation. --- ## Important caveat This still does not mean: “there are currently 5 active runs waiting right now” It means: “5 matching events occurred in the last 60 seconds” Those runs may already have moved on by the time the alert fires. So this is an event rate / backlog onset detector, not a true “current queue depth” monitor. --- ## If you want “currently waiting” semantics That’s harder with plain event triggers. Event triggers are based on events observed over a time window, not current state inventory. So if you want: - current number of waiting runs - current number of pending runs - current queue depth you generally need either: 1. a metric that represents that directly, or 2. a custom monitoring flow that queries the API and alerts on counts For 
AwaitingConcurrencySlot
, a custom admin flow is often the cleanest if you need precise current state counts. --- ## Why your original alert may fire even when nothing is awaiting now Because the automation is not checking “what exists now”. It is evaluating whether the trigger condition was met according to event-bucket semantics over the window, and with 
Proactive
that logic is inverted from your intent. So “no runs awaiting right now” does not imply the automation won’t fire. --- ## Recommendation ### For simple built-in automation Use this:
Copy code
yaml
- name: admin-concurrency-backlog
  description: Notify when many flow runs enter AwaitingConcurrencySlot
  enabled: true
  trigger:
    type: event
    posture: Reactive
    expect:
      - prefect.flow-run.AwaitingConcurrencySlot
    threshold: 5
    within: 60
  actions:
    - type: send-notification
      block_document_id: PLACEHOLDER
      subject: ":rotating_light: Concurrency backlog detected"
      body: |
        5 or more flow runs entered AwaitingConcurrencySlot within 60 seconds.
### If you want per-limit alerts
You’d need to group on a related concurrency-limit resource if that relationship is present in the emitted events. That can work in some cases, but I would verify the event payloads first before relying on it. ### If you want true “current backlog” Build a small monitoring flow that periodically: - queries flow runs in 
AwaitingConcurrencySlot
- groups by deployment / tag / concurrency limit - sends alert if count exceeds threshold That’s much more reliable for operator dashboards/alerts. --- ## Short version Your automation is wrong because: - 
Proactive
is the wrong posture - 
for_each: prefect.resource.id
groups by run id, not backlog - event triggers count events in windows, not current waiting runs Use:
Copy code
yaml
posture: Reactive
and remove:
Copy code
yaml
for_each:
  - prefect.resource.id
--- If you want, I can give you two corrected versions: 1. event-based backlog alert via 
automations.yaml
2. exact current-count monitoring flow for 
AwaitingConcurrencySlot
and 
Pending
runs
r
@Marvin Modify this automation to only trigger if a flow run does not have the tag "poll" - name: admin-flows-long-running description: "Notify Webex when a flow run remains in Running for more than 2 hours." enabled: true trigger: type: event posture: Proactive after: - prefect.flow-run.Running expect: - prefect.flow-run.Completed - prefect.flow-run.Failed - prefect.flow-run.Crashed - prefect.flow-run.TimedOut - prefect.flow-run.Cancelled for_each: - prefect.resource.id match: prefect.resource.id: prefect.flow-run.* match_related: - prefect.resource.role: work-pool prefect.resource.name: - default - haircut threshold: 1 within: 7200 # two hours in secs actions: - type: send-notification block_document_id: PLACEHOLDER subject: "ℹ️Long running flow: {{ flow.name }}" body: | A flow run has been running for more than 2 hours Flow run [{{ flow.name }}/{{ flow_run.name }}]({{ flow_run|ui_url }}) observed in state 
{{ flow_run.state.name }}
.
m
✅ thought for 272.3 seconds
Short answer: you can’t reliably express “does not have tag `poll`” in 
automations.yaml
tag matching alone. In Prefect 3.7.2, tags are exposed as related resources like: - 
prefect.resource.id: prefect.tag.poll
- 
prefect.resource.role: tag
So including a tag is easy with 
match_related
, but excluding a tag is not reliable because related-resource matching is existential/any-match based. A negation like 
!prefect.tag.poll
can still match some other tag on the same flow run. ## What would work for inclusion If you wanted only flows tagged 
poll
, it would look like:
Copy code
yaml
match_related:
  - prefect.resource.role: tag
    prefect.resource.id: prefect.tag.poll
But the inverse is not safe for exclusion. --- ## So for your exact ask If you want “long running, but not tag `poll`”, I would not recommend trying to encode that in 
automations.yaml
directly. ### Best practical options #### Option 1: Put 
poll
flows in a separate work pool / queue Since you’re already filtering by related work-pool, separating polling workloads operationally is the cleanest option. #### Option 2: Add a dedicated tag-aware monitoring flow Use a scheduled admin flow that: - queries running flow runs - excludes runs with tag 
poll
- alerts when runtime > 2 hours This gives exact behavior. #### Option 3: Filter in the notification receiver If your Webex endpoint is custom and can inspect payload details, you could suppress alerts there. But that’s outside Prefect automation logic. --- ## One thing you can do: include only specific tags If your non-poll flows share some positive tag like 
prod
, 
etl
, 
admin-monitored
, etc., then filter for that positive tag instead of trying to exclude 
poll
. Example:
Copy code
yaml
match_related:
  - prefect.resource.role: work-pool
    prefect.resource.name:
      - default
      - haircut
  - prefect.resource.role: tag
    prefect.resource.id:
      - prefect.tag.admin-monitored
That’s reliable. --- ## Your automation, unchanged except for formatting cleanup Your current automation is basically fine structurally:
Copy code
yaml
- name: admin-flows-long-running
  description: "Notify Webex when a flow run remains in Running for more than 2 hours."
  enabled: true
  trigger:
    type: event
    posture: Proactive
    after:
      - prefect.flow-run.Running
    expect:
      - prefect.flow-run.Completed
      - prefect.flow-run.Failed
      - prefect.flow-run.Crashed
      - prefect.flow-run.TimedOut
      - prefect.flow-run.Cancelled
    for_each:
      - prefect.resource.id
    match:
      prefect.resource.id:
        - prefect.flow-run.*
    match_related:
      - prefect.resource.role: work-pool
        prefect.resource.name:
          - default
          - haircut
    threshold: 1
    within: 7200
  actions:
    - type: send-notification
      block_document_id: PLACEHOLDER
      subject: ":information_source:Long running flow: {{ flow.name }}"
      body: |
        A flow run has been running for more than 2 hours
        Flow run <{{ flow_run|ui_url }}|{{ flow.name }}/{{ flow_run.name }}> observed in state `{{ flow_run.state.name }}`.
But I would not add a 
!prefect.tag.poll
filter and trust it. --- ## Recommendation If you want, I can give you a small Prefect admin flow that exactly implements: - find running flow runs older than 2 hours - exclude any run with tag 
poll
- optionally filter to work pools 
default
and 
haircut
- send Webex notification That would be the reliable way to do this.
r
@Marvin ok, rewrite it so it works to exclude anything in a work-queue called "poll"
m
✅ thought for 34.6 seconds
Yep — excluding by work queue is much more workable. Since your automation is already using 
match_related
, you can scope it to only runs related to the work pools you want and exclude the 
poll
queue by matching only the queues you do want. ## Recommended rewrite If you know the non-
poll
queues you want, the safest approach is to positively include them instead of trying to negate 
poll
. For example, if your valid queues are 
default
and 
haircut
and you want to exclude queue `poll`:
Copy code
yaml
- name: admin-flows-long-running
  description: "Notify Webex when a flow run remains in Running for more than 2 hours."
  enabled: true
  trigger:
    type: event
    posture: Proactive
    after:
      - prefect.flow-run.Running
    expect:
      - prefect.flow-run.Completed
      - prefect.flow-run.Failed
      - prefect.flow-run.Crashed
      - prefect.flow-run.TimedOut
      - prefect.flow-run.Cancelled
    for_each:
      - prefect.resource.id
    match:
      prefect.resource.id:
        - prefect.flow-run.*
    match_related:
      - prefect.resource.role: work-pool
        prefect.resource.name:
          - default
          - haircut
      - prefect.resource.role: work-queue
        prefect.resource.name:
          - default
          - haircut
    threshold: 1
    within: 7200
  actions:
    - type: send-notification
      block_document_id: PLACEHOLDER
      subject: ":information_source: Long running flow: {{ flow.name }}"
      body: |
        A flow run has been running for more than 2 hours.
        Flow run <{{ flow_run|ui_url }}|{{ flow.name }}/{{ flow_run.name }}> observed in state `{{ flow_run.state.name }}`.
## If you specifically want “anything except work queue `poll`” There isn’t a reliable “NOT this related resource” pattern I’d recommend here either, for the same general reason as tag exclusion. So instead of trying this:
Copy code
yaml
prefect.resource.name: "!poll"
I’d strongly recommend allowlisting the queues you want monitored. ## If you only need to exclude 
poll
and include everything else Then you have two practical choices: ### 1. Best option: enumerate queues to include Example: - 
default
- 
haircut
- 
batch
- 
priority
### 2. Split your deployments / automations by queue Create one long-running automation for the queues you care about, and simply don’t create one for 
poll
. --- ## Important note about your current config Right now you are filtering work pools here:
Copy code
yaml
match_related:
  - prefect.resource.role: work-pool
    prefect.resource.name:
      - default
      - haircut
If 
default
and 
haircut
are actually work queue names rather than work pool names, then this should be changed to 
work-queue
instead. For example, if those are queue names, use:
Copy code
yaml
match_related:
  - prefect.resource.role: work-queue
    prefect.resource.name:
      - default
      - haircut
If you want, I can rewrite this in one of two exact forms: 1. filter to specific work queues only 2. filter to specific work pool + specific work queues
r
@Marvin filter to specific work pools + specific work queues
m
✅ thought for 9.6 seconds
Got it — here’s the version filtered to specific work pools and specific work queues. This will only apply to flow runs associated with: - work pools 
default
or 
haircut
- and - work queues 
default
or 
haircut
Copy code
yaml
- name: admin-flows-long-running
  description: "Notify Webex when a flow run remains in Running for more than 2 hours."
  enabled: true
  trigger:
    type: event
    posture: Proactive
    after:
      - prefect.flow-run.Running
    expect:
      - prefect.flow-run.Completed
      - prefect.flow-run.Failed
      - prefect.flow-run.Crashed
      - prefect.flow-run.TimedOut
      - prefect.flow-run.Cancelled
    for_each:
      - prefect.resource.id
    match:
      prefect.resource.id:
        - prefect.flow-run.*
    match_related:
      - prefect.resource.role: work-pool
        prefect.resource.name:
          - default
          - haircut
      - prefect.resource.role: work-queue
        prefect.resource.name:
          - default
          - haircut
    threshold: 1
    within: 7200
  actions:
    - type: send-notification
      block_document_id: PLACEHOLDER
      subject: ":information_source: Long running flow: {{ flow.name }}"
      body: |
        A flow run has been running for more than 2 hours.
        Flow run <{{ flow_run|ui_url }}|{{ flow.name }}/{{ flow_run.name }}> observed in state `{{ flow_run.state.name }}`.
## Important behavior note 
match_related
entries are evaluated as related-resource matches, but this structure is the correct way to constrain the automation to runs associated with those pools and queues. If you want to exclude 
poll
, just don’t include it in either list. ## If you want, I can also give you: - a warning version at 1 hour and critical version at 2 hours - a version with cleaner notification text including pool/queue names - a full operator 
automations.yaml
section with this normalized consistently
r
@Marvin ok do a warning and critical version, but only if you can do it in one automation. otherwise the warning version is fine. also indent the automation, i.e. it will be one of many in an automations.yaml And yes, the cleaner notifications text with pool and queue names is good.
m
✅ thought for 18.4 seconds
You’ll want two automations for warning vs critical. I don’t think there’s a clean way to do different thresholds/severities in a single Prefect automation with different notification bodies based on elapsed time. So here’s the recommended two-automation version, indented under 
automations:
for direct use in 
automations.yaml
.
Copy code
yaml
automations:
  - name: admin-flows-long-running-warning
    description: "Notify Webex when a flow run remains in Running for more than 1 hour."
    enabled: true
    trigger:
      type: event
      posture: Proactive
      after:
        - prefect.flow-run.Running
      expect:
        - prefect.flow-run.Completed
        - prefect.flow-run.Failed
        - prefect.flow-run.Crashed
        - prefect.flow-run.TimedOut
        - prefect.flow-run.Cancelled
      for_each:
        - prefect.resource.id
      match:
        prefect.resource.id:
          - prefect.flow-run.*
      match_related:
        - prefect.resource.role: work-pool
          prefect.resource.name:
            - default
            - haircut
        - prefect.resource.role: work-queue
          prefect.resource.name:
            - default
            - haircut
      threshold: 1
      within: 3600
    actions:
      - type: send-notification
        block_document_id: PLACEHOLDER
        subject: ":warning: Long running flow (warning): {{ flow.name }}"
        body: |
          A flow run has been in `Running` for more than *1 hour*.

          *Flow:* {{ flow.name }}
          *Flow run:* <{{ flow_run|ui_url }}|{{ flow_run.name }}>
          *State:* {{ flow_run.state.name }}
          *Work pool:* {{ work_pool.name if work_pool else "unknown" }}
          *Work queue:* {{ work_queue.name if work_queue else "unknown" }}

  - name: admin-flows-long-running-critical
    description: "Notify Webex when a flow run remains in Running for more than 2 hours."
    enabled: true
    trigger:
      type: event
      posture: Proactive
      after:
        - prefect.flow-run.Running
      expect:
        - prefect.flow-run.Completed
        - prefect.flow-run.Failed
        - prefect.flow-run.Crashed
        - prefect.flow-run.TimedOut
        - prefect.flow-run.Cancelled
      for_each:
        - prefect.resource.id
      match:
        prefect.resource.id:
          - prefect.flow-run.*
      match_related:
        - prefect.resource.role: work-pool
          prefect.resource.name:
            - default
            - haircut
        - prefect.resource.role: work-queue
          prefect.resource.name:
            - default
            - haircut
      threshold: 1
      within: 7200
    actions:
      - type: send-notification
        block_document_id: PLACEHOLDER
        subject: ":rotating_light: Long running flow (critical): {{ flow.name }}"
        body: |
          A flow run has been in `Running` for more than *2 hours*.

          *Flow:* {{ flow.name }}
          *Flow run:* <{{ flow_run|ui_url }}|{{ flow_run.name }}>
          *State:* {{ flow_run.state.name }}
          *Work pool:* {{ work_pool.name if work_pool else "unknown" }}
          *Work queue:* {{ work_queue.name if work_queue else "unknown" }}
## One caution on 
match_related
This is the right structure to target specific pools and queues, but in practice I’d still validate one example in your environment after loading it, since related-resource matching depends on those relationships being present on the emitted flow-run events. ## If you only want the warning version Here it is by itself: ```yaml automations: - name: admin-flows-long-running-warning description: "Notify Webex when a flow run remains in Running for more than 1 hour." enabled: true trigger: type: event posture: Proactive after: - prefect.flow-run.Running expect: - prefect.flow-run.Completed - prefect.flow-run.Failed - prefect.flow-run.Crashed - prefect.flow-run.TimedOut
- prefect.flow-run.Cancelled for_each: - prefect.resource.id match: prefect.resource.id: - prefect.flow-run.* match_related: - prefect.resource.role: work-pool prefect.resource.name: - default - haircut - prefect.resource.role: work-queue prefect.resource.name: - default - haircut threshold: 1 within: 3600 actions: - type: send-notification block_document_id: PLACEHOLDER subject: "⚠️ Long running flow (warning): {{ flow.name }}" body: | A flow run has been in 
Running
for more than 1 hour. Flow: {{ flow.name }} Flow run: <{{ flow_run|ui_url }}|{{ flow_run.name }}> State: {{ flow_run.state.name }} Work pool: {{ work_pool.name if work_pool else "unknown" }} Work queue: {{ work_queue.name if work_queue else "unknown" }} ``` If you want, I can also tighten these so the critical alert won’t duplicate the warning one too noisily, e.g. by changing wording, adding escalation hints, or suggesting a follow-up remediation automation.
3 Views
Open in Slack
PreviousNext
Powered by