@Marvin how can I automatically retry Crashed flow

To handle retries for flows that crash, you can utilize retry policies within tasks. Specifically, you can set the

retries

and

retry_delay_seconds

parameters on a task to specify how many times it should retry and the delay between retries. Here's a conceptual example:

python
import httpx
from prefect import flow, task

@task(retries=2, retry_delay_seconds=10)
def potentially_crashing_task():
    # Your task logic here
    ...

@flow
def my_flow():
    potentially_crashing_task()

my_flow()

In this case, if

potentially_crashing_task

crashes, Prefect will automatically retry the task up to 2 times, with a delay of 10 seconds between retries. Note that retries do not create new task runs but add a new state to the state history of the original task run. If you are working with dbt Cloud specifically, there is a function in the

prefect-dbt

library called

run_dbt_cloud_job

that includes a

targeted_retries

parameter for retrying failed steps. The documentation can be found here. For more detailed information on handling retries and tasks in Prefect, you can check the Tasks concept documentation.

@Marvin the retries parameter does not work with Crashed flows. How can I retry using Automations?

To automate the retry of crashed flows, you can use Prefect Automations. Automations allow you to specify triggers and actions based on different conditions or events. One way to address this is by setting up a trigger for a flow run state change. For instance, you might expect your production flows to complete within a certain time frame. If a flow run gets stuck in the 'Running' state due to infrastructure or network issues, you could configure an Automation to take action if it exceeds that time frame. This action could include restarting the flow or notifying a team member to take manual remediation steps. Currently, you can create Automations through the Prefect UI or programmatically via the Automations API. These Automations can be reactive in response to events or proactive in the absence of an expected event. For more information, you should refer to the Automations concept documentation. Here you can find details on how to set up triggers and what actions you can perform. If you want to dive deeper into creating Automations, consult the Automations guide. Unfortunately, the excerpts do not provide a direct code example for this automation. If you need help with specific steps in setting up Automations for your use case, it might be helpful to search or ask in the Prefect community.