Idriss Bellil
10/28/2025, 1:23 PMMarvin
10/28/2025, 1:23 PMMarvin
10/28/2025, 1:27 PMretries only cover normal failures (exceptions), not infrastructure “Crashed” states. To handle crashes, use an Automation (Prefect Cloud) to rerun the flow run.
- You can still keep @flow(retries=1, ...) for non-crash failures; combine both for full coverage.
If you’re on Prefect Cloud (recommended approach)
- Create an Automation:
- Trigger: “Flow run enters state” → Crashed
- Optional filters: limit to a specific deployment or tag to scope it
- Action: “Rerun flow run”
- Guard/limit so it only retries once: add a condition that only triggers when the flow run’s attempt/run count indicates it hasn’t been rerun yet (i.e., first attempt only). In the UI, this is typically expressed by filtering on the flow run’s “run count/attempt” so the action only fires once per run.
- This gives you a single automatic rerun if the original run crashes. If it crashes again, it won’t keep looping.
Also add a normal failure retry for completeness:
from prefect import flow
@flow(retries=1, retry_delay_seconds=0)
def my_flow():
...
If you’re running OSS only (no Cloud)
- The platform doesn’t include Automations, so the common pattern is a lightweight “watchdog” flow or external process that:
- Lists recent flow runs that are in a Crashed state and have not been retried yet
- Triggers a single rerun (either by rerunning the same run or creating a new flow run with the same deployment/parameters)
- I can share a small example watcher if you confirm your environment.
Quick questions so I can tailor exact steps:
- Are you on Prefect Cloud or OSS?
- Do you want the retry to be a true “rerun” of the same run, or to create a new run with the same parameters?
- Any specific work pool/worker type (Kubernetes, Docker, Process) I should account for?Idriss Bellil
10/28/2025, 2:37 PMMarvin
10/28/2025, 2:37 PMMarvin
10/28/2025, 2:38 PMauto-rerun. Then add a trigger filter to only match flow runs that do not have the auto-rerun tag. That guarantees a single automatic retry.
If you’d like, tell me if you’re on Prefect Cloud and I’ll give you the exact clicks to set up the first-attempt guard in your UI.Idriss Bellil
10/29/2025, 8:46 AMMarvin
10/29/2025, 8:47 AMMarvin
10/29/2025, 8:51 AMauto-rerun, and add a trigger filter so it only fires for flow runs that do not have auto-rerun. This guarantees a single auto-rerun.
If you tell me which fields you see in the Automation builder (UI vs API), I can give you exact clicks/JSON to set the one-time guard.
References:
- Automations overview
- Event triggersIdriss Bellil
10/29/2025, 8:56 AMMarvin
10/29/2025, 8:56 AMMarvin
10/29/2025, 8:57 AMauto-rerun
- Action: Run the same deployment, copy parameters from the triggering run, and add tag auto-rerun to the new run
- Result: the first crash creates a tagged rerun; if that rerun crashes, it’s ignored because it has the tag.
Docs for reference:
- Automations overview
- Event triggers
If you tell me which fields you see in the UI (e.g., can you filter on “attempt” or “tags”? and can you “copy parameters from triggering run”?), I’ll give you exact click-by-click steps.Idriss Bellil
10/31/2025, 9:14 AMMarvin
10/31/2025, 9:14 AMMarvin
10/31/2025, 9:16 AMauto-rerun so you only get pinged if the retried run also crashes/fails.
Docs:
- Automations overview
- Event triggers
2) About the “1 per hour/day” settings and preventing infinite retries
- If your retry automation uses “Change flow run state to Scheduled”, a simple “1 time within 1 hour” does NOT cap it to one retry. Every time that same run crashes again, it’ll fire again. That’s why it can loop.
- To guarantee a single retry, the most robust approach is:
- Use the “Run deployment” action to create a new run and add a tag like auto-rerun
- Add a trigger filter so it only fires for runs that do NOT have the auto-rerun tag
- Copy parameters from the triggering run
- Result: each original crashed run produces exactly one tagged rerun; if the rerun crashes, it won’t be retried again
If you prefer to keep “Change state to Scheduled” (same run) and still limit to one retry per run, there isn’t a perfect guard based on time windows. You’d need a per-run marker (which the ChangeState action can’t add). In that case, the safer notification approach above (Proactive “expect recovery”) keeps noise down while you allow one reschedule on every crash.
Want exact click-by-click steps? Let me know:
- Whether your retry automation is “Change state to Scheduled” or “Run deployment”
- If your UI shows filters for tags on the trigger and lets you “copy parameters from triggering run” in the action
I’ll tailor the configuration for your setup so you get a single auto-retry and clean notifications.