@Marvin is there a better way of handling flow runs that still show in the dashboard as "running" but arent actually running? I'm seeing flow runs in "running" state from days ago that are definitely not running. can I set a max timeout for flows or something?

It appears that you're experiencing an issue with flow runs showing as "running" in the Prefect dashboard even though they aren't actually running. This can be frustrating, but there are ways to handle it. 1. Flow Run Timeout: Prefect provides a mechanism to handle timeouts for flow runs. You can define a timeout for your flow runs, and if a flow exceeds this timeout, a

FlowRunTimeoutError

is raised. You can implement this in your flow configuration to ensure that long-running flows are automatically stopped and marked as failed if they exceed the specified duration. 2. State Management: Prefect states are rich objects that provide information about the status of a flow or task run. You might want to explore the state management features of Prefect to programmatically update or transition states if you detect that a flow is not running as expected. You can refer to the following resources for more details: - Flow Engine Source Code which includes handling for flow run timeouts. - Manage States in Prefect for insights on state management and how to handle different states. These resources should help you configure your flows to better handle unexpected states in the dashboard.

hi @chris! you can delete them if you want you can set timeouts! you can also setup a deployment that has a trigger which will

expect

a

prefect.flow-run.Completed

event

after

prefect.flow-run.Running

event

within

some amount of time like

3600

and it can use the prefect client to call the

delete_flow_runs

client method with a filter

To handle flow runs that remain in the "running" state, you can indeed set up mechanisms in Prefect to address this. Here's how you can approach it: 1. Flow Run Timeout: You can define a timeout for your flow runs. If a flow exceeds this timeout, a

FlowRunTimeoutError

is raised. This can be handled within your flow logic to transition the flow to a terminal state. 2. Event-Based Triggers: You can set up a deployment with a trigger that expects a

prefect.flow-run.Completed

event after a

prefect.flow-run.Running

event within a specified timeframe (e.g., 3600 seconds). If the expected event does not occur within the timeframe, you can take corrective actions such as terminating the run. 3. Using Prefect Client: You can utilize the Prefect client to programmatically manage flow runs. For instance, the

delete_flow_runs

client method can be used to delete flow runs based on specific filters. This can be part of a cleanup process triggered by the absence of a completion event. For more details, you can refer to: - Automations and Triggers in Prefect for setting up event-based triggers. - Flow Engine and Timeout Handling for understanding how timeouts are managed. These strategies should help you manage flow runs that are inaccurately marked as "running" in the dashboard.

this usually happens when infra disappears unexpectedly. perhaps we should have a sane default set of automations like "delete flow runs after N hours" but we generally try to avoid doing unexpected things

gotcha, appreciate the feedback. definitely agree it makes sense to not make that a default. not sure what compelled me to do this but I looked you up and - go blue!!

ayyy go blue!