Hey all has anybody had success in detecting hanging flows I

Question

Hey all has anybody had success in detecting hanging flows I have the following automation set up on prefect cloud Prefect v2 8 7 but my flow ran for 6 5 hours this morning and nothing was sent to our slack channel to notify us 2nd time this has happened Is there a work around others use to detect hanging flows Or am I missing something on the automation side Flow run id 6c1ee67b db06 45ea 8a6a 9585f7cf1d52 Automation id 5d633fec fb09 409f b1fa 5270dcc429f2

Chris Guidry · Answer

Hi Zach taking a look now

Chris Guidry · Answer

Ahhh yes sorry this is a common source of mixups when using the Stays In aka `Proactive` automations So your automation here says If a flow run emits any one of the `Resuming Running Pending Late` events and then doesn t emit another event for the next 4 hours trigger the actions The problem here comes from including both `Pending` and `Running` in the same automation most flow runs generally go from `Pending` to `Running` quite quickly so the `Running` event clears out the automation after it started from the `Pending` event Here s the sequence 07 00 05 518905 UTC gt `Pending` The automation starts watching for subsequent events 07 00 05 705739 UTC gt `Running` The automation says cool we got an event so it s not stuck in `Pending` then closes itself out You may want to create a second automation that just catching things stuck in `Pending` separately from the `Running Resuming Late` ones

Zachary Loertscher · Answer

Ah I see so once it has transitioned from `Pending` gt `some other state` the automation stops watching for hangs I ll remove Pending from that list in that case then Unfortunately we are at our 3 automation limit so I cannot add another automation Sounds like there currently isn t a way for the automation to continue monitoring even when a status changes Or maybe watching multiple statuses in a single automation isn t yet supported

Chris Guidry · Answer

Sorry yes that s accurate an automation will only start based on any one of those events and then clear when it reaches one of the expected events in this case `prefect flow run ` any flow run event Another way you can approach this is to switch to a custom trigger not using the UI and use something like this ```after prefect flow run Pending prefect flow run Running prefect flow run Resuming prefect flow run Late expect prefect flow run Completed prefect flow run Crashed prefect flow run Failed ``` Where instead of looking for any event you set it to look for final events So it could start with any of those pending running etc and will only clear if it gets to completed crashed failed etc within 4 hours I ll follow up in DM about the 3 automation limit

Zachary Loertscher · Answer

Awesome this is great I ll switch this over to a custom trigger in that case raised hands