<@ULVA73B9P> I need a way to automatically retry c...
# ask-marvin
i
@Marvin I need a way to automatically retry crashed flows once
m
thought for 256.3 seconds
Got it — you want crashed flow runs to automatically retry once. A couple quick notes: - In Prefect 3.x, flow/task
retries
only cover normal failures (exceptions), not infrastructure “Crashed” states. To handle crashes, use an Automation (Prefect Cloud) to rerun the flow run. - You can still keep
@flow(retries=1, ...)
for non-crash failures; combine both for full coverage. If you’re on Prefect Cloud (recommended approach) - Create an Automation: - Trigger: “Flow run enters state” → Crashed - Optional filters: limit to a specific deployment or tag to scope it - Action: “Rerun flow run” - Guard/limit so it only retries once: add a condition that only triggers when the flow run’s attempt/run count indicates it hasn’t been rerun yet (i.e., first attempt only). In the UI, this is typically expressed by filtering on the flow run’s “run count/attempt” so the action only fires once per run. - This gives you a single automatic rerun if the original run crashes. If it crashes again, it won’t keep looping. Also add a normal failure retry for completeness:
Copy code
from prefect import flow

@flow(retries=1, retry_delay_seconds=0)
def my_flow():
    ...
If you’re running OSS only (no Cloud) - The platform doesn’t include Automations, so the common pattern is a lightweight “watchdog” flow or external process that: - Lists recent flow runs that are in a Crashed state and have not been retried yet - Triggers a single rerun (either by rerunning the same run or creating a new flow run with the same deployment/parameters) - I can share a small example watcher if you confirm your environment. Quick questions so I can tailor exact steps: - Are you on Prefect Cloud or OSS? - Do you want the retry to be a true “rerun” of the same run, or to create a new run with the same parameters? - Any specific work pool/worker type (Kubernetes, Docker, Process) I should account for?
i
@Marvin the automation will do, will it keep retrying the crashed flow forever or I have a way to make it react only the first time?
m
thought for 71.7 seconds
Short answer: it won’t retry forever if you add a guard. You can make the automation react only the first time a run crashes. Two common ways to limit it to a single auto-rerun: - First-attempt filter (recommended): In the Automation builder, set the trigger to “Flow run enters state = Crashed” and add a condition that matches only the first attempt/run of that flow run (i.e., first crash). This ensures the “Rerun flow run” action fires once and won’t keep looping if the rerun crashes again. - Tag-based guard (alternative): Instead of “Rerun flow run,” use “Run a deployment” to start a new run with the same deployment and add a tag like
auto-rerun
. Then add a trigger filter to only match flow runs that do not have the
auto-rerun
tag. That guarantees a single automatic retry. If you’d like, tell me if you’re on Prefect Cloud and I’ll give you the exact clicks to set up the first-attempt guard in your UI.
i
@Marvin I see a template for the automations to retry crashed flows, the only action I see there is to change the state to Scheduled, should I assume this will be enough (the automation changing the state to Scheduled)? or I should add some other action in there to express retry?
m
thought for 264.6 seconds
Short answer: yes — “Change flow run state to Scheduled” is the correct action and is sufficient to retry the same run. When the run is put back into Scheduled, a worker will pick it up and execute it again. Important: if you only add that action with a simple “Crashed” trigger, it will reschedule every time the run crashes, which can loop. To make it fire only once, add a guard. Two practical patterns: - If your UI exposes an “attempt/run count” filter: add a condition so it only triggers on the first crash attempt for that flow run. That gives you a single automatic retry. - Tag-based guard (works everywhere): instead of changing the state, use the “Run deployment” action to start a new run with a tag like
auto-rerun
, and add a trigger filter so it only fires for flow runs that do not have
auto-rerun
. This guarantees a single auto-rerun. If you tell me which fields you see in the Automation builder (UI vs API), I can give you exact clicks/JSON to set the one-time guard. References: - Automations overview - Event triggers
i
@Marvin good point, in the UI in the trigger part, I see y time within x [days,hours,..], so I guess if I set 1 time within 1 hours, it should be enough for preventing such infinite retry loop, am I right?
m
thought for 55.7 seconds
Short answer: no. “1 time within 1 hour” does not cap it at one retry. With threshold=1, the automation will fire every time a Crashed event occurs (per “for each” resource), so if the rerun crashes again within that hour it will trigger again and can loop. To make it one-and-done, use one of these guards: - If your UI exposes an “attempt/run count” filter: add a condition so it only triggers on the first attempt/crash for that flow run. Then “Change state to Scheduled” is fine and will only happen once. - Recommended when that filter isn’t available: switch the action to “Run deployment” and add a tag guard. - Trigger: Flow run enters Crashed - Filter: Flow run does NOT have tag
auto-rerun
- Action: Run the same deployment, copy parameters from the triggering run, and add tag
auto-rerun
to the new run - Result: the first crash creates a tagged rerun; if that rerun crashes, it’s ignored because it has the tag. Docs for reference: - Automations overview - Event triggers If you tell me which fields you see in the UI (e.g., can you filter on “attempt” or “tags”? and can you “copy parameters from triggering run”?), I’ll give you exact click-by-click steps.
i
@Marvin the automation worked nicely but now I have a related issue. I have another automation that notifies on Failed & Crashed flows, so I am seeing both automations triggering now, that retry once a day & the notification. 2 questions now: 1. how to not get notified about crashed and retried flow? I could think about one way but it could be crappy, like make the notification only trigger after 10min while in the state of Crashed & Failed, but I wonder if you have something better. 2. I don't like that I am setting it up like 1 time per hour/day/.. as I don't understand it fully, so if my flows run a couple times per day, then I guess setting the automation once per hour makes more sense right? because I want all crashed flows to be retried once only (so the one hour is kinda the cool down period and I don't get infinite retries
m
thought for 87.8 seconds
Great questions — here’s how I’d set this up to avoid noisy alerts and make the retry deterministic. 1) Avoid notifying for crashes that are auto-retried Best practice is to turn your notification automation into a “wait-and-see” (Proactive) alert: - Trigger: Flow run enters Crashed or Failed - Posture: Proactive - Expect: The run enters Scheduled or Running (i.e., it’s retried) within X minutes - Within: e.g., 5–10 minutes - Action: Send notification only if the expectation is not met Result: If your retry automation reschedules/re-runs the flow quickly, no alert is sent. If the run does not recover (no retry starts) within the window, you get notified. Alternative patterns: - Notify only on the second terminal event: - Reactive, threshold=2 within e.g. 1 day, for_each per flow-run ID, states Crashed/Failed. This alerts only if the same run terminally fails twice (works if you “Change state to Scheduled” on the same run; won’t work if you create a new run). - If you use the “Run deployment” + tag guard retry pattern (recommended below), send notifications only for runs with tag
auto-rerun
so you only get pinged if the retried run also crashes/fails. Docs: - Automations overview - Event triggers 2) About the “1 per hour/day” settings and preventing infinite retries - If your retry automation uses “Change flow run state to Scheduled”, a simple “1 time within 1 hour” does NOT cap it to one retry. Every time that same run crashes again, it’ll fire again. That’s why it can loop. - To guarantee a single retry, the most robust approach is: - Use the “Run deployment” action to create a new run and add a tag like
auto-rerun
- Add a trigger filter so it only fires for runs that do NOT have the
auto-rerun
tag - Copy parameters from the triggering run - Result: each original crashed run produces exactly one tagged rerun; if the rerun crashes, it won’t be retried again If you prefer to keep “Change state to Scheduled” (same run) and still limit to one retry per run, there isn’t a perfect guard based on time windows. You’d need a per-run marker (which the ChangeState action can’t add). In that case, the safer notification approach above (Proactive “expect recovery”) keeps noise down while you allow one reschedule on every crash. Want exact click-by-click steps? Let me know: - Whether your retry automation is “Change state to Scheduled” or “Run deployment” - If your UI shows filters for tags on the trigger and lets you “copy parameters from triggering run” in the action I’ll tailor the configuration for your setup so you get a single auto-retry and clean notifications.