<@ULVA73B9P> What dictates a Work Pool's Status? C...
# ask-marvin
t
@Marvin What dictates a Work Pool's Status? Currently there's 90 late runs pending but it is still "Ready"
m
thought for 223.0 seconds
Short answer: A Work Pool’s Status is about worker connectivity and pause state, not backlog. In Prefect 3.x: - Ready: The pool is not paused and has been polled by at least one worker recently (roughly within the last minute). - Not Ready: The pool is not paused but hasn’t been polled by any worker recently. - Paused: The pool is manually paused and will not hand out new work. So having “90 late runs pending” does not change the pool’s Status; it can still show Ready if a worker is actively polling it. Why you might see many late runs even when Ready: - Worker is polling but constrained by concurrency limits (pool, deployment, or tag-based). - Queue filters don’t match your runs (e.g., workers filtering to tags/queues your runs don’t have). - Deployment is paused or has zero concurrency limit. - Worker can’t provision infrastructure or is erroring, but still polls. Docs: Work Pools (Prefect 3) If you want, share: - Your work pool type, whether it’s paused, and its concurrency limit - Worker logs (do you see repeated “no eligible flow runs” messages?) - Any deployment or tag concurrency limits … and I can help pinpoint why those runs are late.
t
@Marvin I want to trigger an automation once there’s no more late runs, is that possible?
m
thought for 525.6 seconds
Yes. You can do this with an Automation using a Metric trigger. Two ways to set it up (Prefect 3.x): Option A — Simple metric trigger - Trigger: Metric - Metric: Count of flow runs in the Late state - Scope: Filter to your work pool/queue/deployment/tags as needed - Condition: equals 0 - Optional: require it to be true for N minutes to avoid flapping - Action: whatever you want (e.g., send a notification) Option B — Only fire after there were late runs If you only want it to fire when the backlog clears (i.e., there were late runs and then there are none), use a Sequence trigger: - Step 1: Metric “Late flow run count” > 0 (scoped appropriately) - Step 2: Metric “Late flow run count” == 0 (optionally sustained for N minutes) - Action: your desired action Notes - This is based on the count of Late runs; it’s independent of work pool “status”. - In the UI: Settings → Automations → New Automation → choose Metric (or Sequence) trigger and configure as above. If you share how you want to scope it (work pool vs. specific deployments) and what action you want, I can outline the exact configuration.
t
@Marvin When I create automation all I see is this: When I Observe Don't observe Any event matching prefect.flow-run.Failed From the following resources prefect.flow-run.* Threshold 5 times within Within 1 Duration unit Minutes Evaluation Options Evaluate trigger only after observing an event matching All events Filter for events related to All resources Where can I find metric trigger? I am using prefect 3.0
m
thought for 313.7 seconds
Short answer: In Prefect 3.0, the Automations UI only exposes event-based triggers right now. That’s why you’re seeing “Observe/Don’t observe Any event matching …” — there isn’t a “Metric trigger” option in the UI yet. What you can do today: - UI: Use event triggers (e.g.,
prefect.flow-run.Failed
) with the “Threshold … times within … minutes” options you’re seeing. That covers “N failures in M minutes”-style alerts. - SDK/API: Metric-based automations do exist in the SDK as
MetricTrigger
and
MetricTriggerQuery
, but they’re not configurable in the UI yet. You can create them programmatically. Useful links: - Concepts: Automations &amp; triggers in Prefect 3 - How-to: Creating automations and triggers Note on CLI: There isn’t a CLI command to create automations;
prefect automations
supports listing/inspecting/enabling/disabling/deleting existing automations only. If you share which metric and condition you want (e.g., runtime > X, lateness > Y, success rate < Z), I can draft a concrete SDK snippet to create the corresponding metric-based automation.
t
@Marvin That would be great, I want to trigger an automation if there's no
inference/inference
deploymeny with Late/Pending/Scheduled state, trigger a different deployment and send a notification using slack webhook
m
thought for 646.9 seconds
Great use case. You can do this in Prefect 3.0 today with an Event-based automation in the UI using the “Don’t observe” posture (absence detection), then attach two actions: Run a different deployment and Send a Slack notification. Here’s how to configure it in the UI: 1) Create the trigger - Go to Automations → New Automation - When I: Don’t observe - Any event matching: add these three events -
prefect.flow-run.Scheduled
-
prefect.flow-run.Pending
-
prefect.flow-run.Late
- From the following resources: filter to your deployment - Filter for events related to → Deployment → name equals
inference/inference
- Threshold: 1 - Within: set a window that’s slightly larger than the expected cadence of that deployment (e.g., if it should schedule every 5 minutes, use 10–15 minutes) - Evaluation options: leave defaults unless you need to scope evaluation further This configuration means: “If we do not observe at least one Scheduled/Pending/Late flow-run event for the deployment inference/inference within the chosen window, fire the automation.” 2) Add the actions - Add action → Run deployment → select the fallback deployment you want to kick off - Add action → Send notification - Choose your Slack Webhook block (or create one first under Blocks → Slack Webhook) - Provide a message template (include details like the source deployment, window used, and the triggered deployment) Notes and tips - Pick a “Within” window that matches your expectations. If your deployment should be continuously scheduling, use a short window; if it’s cron-based, use a window larger than the cron interval. - You can add all three events (Scheduled, Pending, Late) in the same trigger as separate rows under “Any event matching.” - This approach is event-driven (absence of events). If you specifically want a metric/count-based check (e.g., “count of currently Scheduled/Pending/Late runs = 0”), that isn’t exposed in the UI yet; it’s possible via the SDK with a Metric-based trigger. I can draft that if you prefer an API-based setup. References - Automations and triggers overview: Automations &amp; triggers in Prefect 3 - How-to guide for creating automations: Creating automations and triggers If you share: - The exact name of the deployment you want to trigger - Your preferred “Within” window - The Slack Webhook block name (or if you need help creating one) I can provide a quick checklist and a suggested notification message body you can paste into the UI. If you want the SDK/API version instead (metric trigger + actions), I can draft that as well.