<@ULVA73B9P> why do I see late run notification in...
# ask-marvin
a
@Marvin why do I see late run notification in slack for the runs where there doesn't exist a flow run in ui. What I noticed is , flow exist but generates a late notice for flowrun id which is not seen in ui. When we go to ui , there is no late runs, I have set late run. For staying in 5 minutes only, still issue exists
m
thought for 298.5 seconds
Thanks for reporting this — I know it’s confusing to get a “late run” Slack notification for a flow run that you can’t find in the UI. Short version: late-run Slack notifications are triggered by events. If a flow run is marked Late and an event is emitted, your automation will fire even if the run is subsequently cancelled, deleted, or otherwise hidden by UI filters before you look it up. That’s why you can see a valid run id in Slack but not find it in the UI afterward. Common causes - The run was cancelled/deleted after the Late event was emitted - Example: someone paused the deployment, an automation cancelled scheduled runs, or code called cancel/delete — the event was already in the queue so the notification still fires. - Event processing delay vs. DB/UI state - Events are asynchronous. A Late event can be processed seconds/minutes after the run has already been cancelled or removed. - UI filters hiding the run - The “Parent runs only” filter can hide subflow/child runs. If the late notification was for a subflow, it won’t show unless you disable that filter. - Wrong workspace or link mismatch - If the automation or link references another workspace/environment, you’ll see the ID but won’t find it where you’re looking. Quick checks to narrow it down 1) Search directly by ID in UI - Paste the flow_run_id from Slack into the UI search box. Also toggle off “Parent runs only” to include subflows. 2) Verify via CLI
Copy code
prefect flow-run inspect <FLOW_RUN_ID>
If it exists, you’ll get details. If it’s been cancelled/deleted, you’ll see that status or a not-found. 3) Check the automation’s history - In Automations > Your automation > History, confirm which event triggered the notification and when. If the event is “prefect.flow-run.Late,” it likely fired just before the run was cancelled or removed. 4) Check for anything that cancels scheduled runs - Deployment paused/unpaused around that time? - Any automations that cancel runs based on conditions? - Concurrency settings that might cancel or reject new runs? - Worker logs around that timestamp showing “marked late” followed by “cancelled” or “deleted”. Ways to reduce false/ghost notices - Add “stays Late for” a bit longer - You mentioned 5 minutes — increasing this to 10–15 minutes can cut down on transient/short-lived lateness. - Scope the automation - Limit to the specific deployment(s) and work pool(s) you care about so you don’t get noise from test runs or subflows you don’t intend to watch. - Make sure you can see subflows - Disable “Parent runs only” when validating, or include flow name/deployment in your Slack template so you know what to look for. References - Automations overview: Automations docs - Events (what triggers these notifications): Events docs If you can share 2–3 example Slack messages (flow_run_id + timestamps) and confirm: - Prefect Cloud or self-hosted? - Workspace name - The exact trigger you used (“Flow run stayed Late for 5 minutes”, or “Flow run stayed Scheduled for 5 minutes”, etc.) - Whether subflows might be involved …I’ll dig in and help pinpoint the exact cause in your setup.