@Marvin How do I re-continue a crashed flow from where it crashed?

To handle a crashed flow and resume it in Prefect, here's what you need to know based on the version of Prefect you're using: 1. Prefect 2.x: Prefect 2.x has no built-in mechanism for automatically resuming a crashed flow from the exact point of failure. However, you can make your flow resilient to failure by leveraging retries and state change hooks. For example, you can configure retries for your tasks, which will automatically retry the tasks upon failure. For more complex logic, you might need to implement custom mechanisms using state change hooks. - Retries Example:

python
     from prefect import flow, task
     
     @task(retries=3, retry_delay_seconds=10)
     def my_task():
         # task logic here
         pass
     
     @flow
     def my_flow():
         my_task()

- State Change Hooks: They can help ensure that cleanup and necessary state changes are handled if a crash occurs. You might need to implement more sophisticated state management to resume from failure points. - Documentation for retries and state change hooks: Prefect 2.x State Change Hooks 2. Prefect 3.x: Prefect 3.x provides more direct control through

pause_flow_run

and

resume_flow_run

functions. The

suspend_flow_run

method can be used to pause the flow run, which can be resumed later using

resume_flow_run

. - Pausing and Resuming Example:

python
     from prefect import task, flow, pause_flow_run, resume_flow_run
     
     @task
     def my_task():
         # task logic here
         pass
     
     @flow
     def my_flow():
         my_task()
         pause_flow_run()  # Pauses the flow execution
         # Additional logic to handle resuming
         resume_flow_run()  # Call this to resume the flow

The flow will exit and tear down its infrastructure upon pausing. When resumed, the flow execution will start from the beginning, but you can use task caching to avoid re-computation of completed tasks. - Documentation: Prefect 3.x Pause and Resume Based on your specific use case and version of Prefect, you can employ these strategies to handle and resume flows after crashes.