We need alerting when a flow is submitted for execution but

Question

We need alerting when a flow is submitted for execution but is never executed Can we create an automation for this Basically for every flow send an email if a flow is submitted but not started after 60 seconds Is this possible

Kevin Kho · Answer

I actually think there is none because there is only ` Does not start ` and this refers to something not leaving a Scheduled state I think we d need an issue for this

Kevin Kho · Answer

< Marvin> open Feature Request Automation for Flow stuck in Submitted

Marvin · Answer

Matthew Seligson · Answer

What other solutions are there to this problem We have state handlers that alert when a flow fails but no coverage for cases where a flow doesn t even start

Kevin Kho · Answer

I am not seeing one at the moment myself let me ask the team for ideas

Anna Geller · Answer

< Matthew Seligson> this is what Lazarus is for If the flow run stays in the Submitted state for longer than 10+ minutes Lazarus rescheduled it up to 3 times There is nothing you have to do on your end to handle that and I don t think we need Automation for this

Anna Geller · Answer

Anna Geller · Answer

In general this indicates an issue with your execution layer e g your agent can t deploy your flow run to a given execution layer something is wrong in your Kubernetes or ECS cluster `Submitted` state means that the agent was able to pick up the scheduled flow run and it submitted it for execution but something has gone wrong in the flow run deployment Here are some steps you may take to investigate it further check the agent logs to see if anything suspicious stands out there running an agent with the ` show flow logs` option can show more useful output verify that your execution layer is able to pull your flow run s image e g if the image needs to be pulled from a container registry make sure your container can reach the Internet and has appropriate permissions to pull the image verify that your execution layer has enough permissions to spin up all required resources For instance Kubernetes agent needs proper RBAC permissions to deploy your flow run as a Kubernetes job ECS agent needs custom `task execution role` with permissions to describe VPCs register a task definition and run the ECS task verify that your execution layer has enough capacity on the cluster to deploy your flow run

Matthew Seligson · Answer

Thanks < Anna Geller> We don t want to reschedule it we just want to know that it failed to execute a flow As you mentioned the agent is picking up the flow but something is going wrong in the flow run deployment Looking at logs is certainly helpful for a post mortem but they are not the same as an alert we d like to be notified when the flow run is submitted but not running

Anna Geller · Answer

in that case perhaps you can create a custom flow level state handler in general it would be better to tackle the root cause of the issue Do you know why your infrastructure cannot deploy the flow run

Anna Geller · Answer

also just to be 100 sure you are on Prefect Cloud right

Matthew Seligson · Answer

Thanks Anna We have a flow state handler but it is not triggered if the flow isn t started The issue is between the submitted and running state Yes we are on Prefect Cloud

Anna Geller · Answer

Gotcha I can totally understand why getting such an alert is useful but to set expectations I could reopen this issue but I wouldn t expect it to be tackled anytime soon since we already have a dedicated service that handles such issues in 1 0 which is Lazarus and 2 0 is the priority so the best bet would be to tackle the root cause of the issue and prevent those scenarios in the first place e g by allocating more resources and monitoring such use cases on the infrastructure level