<@ULVA73B9P> did Prefect have any update today? an...
# marvin-ai
m
@Marvin did Prefect have any update today? an automation, usually is monitoring all flows stay in pending stage for 15 minutes, but starts from 2 hours ago, this automation is crazy with every finished flows, which means every finished flows triggered this automation. do you have any idea about that? Prefect version 2.xx
m
thought for 95.0 seconds
Thanks for flagging this. I’m not aware of any Prefect Cloud 2.x update in the last few hours that would change Automation behavior, and I don’t see any active incidents. You can double‑check the platform status here: Prefect Status. A few things to consider and some quick checks: What “stayed in a state” actually does - The “Flow run stayed in state = Pending for 15 minutes” trigger fires once when a run has remained Pending continuously for the full 15 minutes. It can fire even if the run later transitions to Running/Completed. So if your worker(s) were busy/offline/constrained for ~15+ minutes, all those runs could have legitimately triggered the automation, then finished afterward—making it look like “finished flows triggered the automation.” Checklist to verify - Pick one of the noisy flow runs and open its timeline. Did it remain Pending for ≥15 minutes before it started? If yes, the automation is behaving as designed. - Check your Work Pool and Worker health: - Is the worker online and heartbeating? - Any recent restarts or image changes? - Any concurrency limits (pool-level or tag-based) that could have queued runs? - Any spike in queued runs/long Pending backlog around two hours ago? - Look at the Automation execution details for a sample alert. The triggering event payload will show state name, duration, and timestamp that satisfied the condition. If alerts truly fired after completion (possible bug) - If you see alerts timestamped after a run had already left Pending (i.e., it was not still Pending at the 15‑minute mark), please share: - Automation ID or name - 2–3 example flow run IDs that incorrectly triggered, with timestamps - Approximate time window (UTC) when this started I can dig in further with that information. Noise‑reduction options - Increase the threshold (e.g., 30–45 minutes) if short backlogs are acceptable. - Scope the automation to specific deployments/tags where Pending is actionable. - If your runs are scheduled, consider a “late to start” style condition (alerts when a scheduled run has not started by its scheduled time), which can be less noisy than generic Pending thresholds. Let me know the automation’s exact configuration (trigger type, state = Pending, duration = 15m, any filters) and share 1–2 example run IDs. I can help confirm whether it’s expected behavior or something we should escalate.
m
this page has a active incident Delayed Automation Triggers Partial Service Disruption Incident Status Partial Service Disruption Components Cloud Automations Locations Prefect Cloud March 19, 2026 03:17 UTC INVESTIGATING We are investigating delays in automation trigger processing. Some automations may experience delayed execution. do you have ability to check what happened?
@Marvin