<@ULVA73B9P> I have an automation that retries cra...
# ask-marvin
i
@Marvin I have an automation that retries crashed flows (generally because of infra failing to commission in AWS ECS Fargate like timeout to fetch ECR image is the most common issue), can I make the automation retry more than once? (example: flow run starting -> crashed -> automation trigger -> scheduled -> Crash again -> exit -- can I replace this with another retry? -- )
m
thought for 125.5 seconds
Short answer: yes. In Prefect 3.x, an automation that reacts to flow-run crashed events can re-trigger multiple times. There isn’t a built-in “only fire once” or “max attempts” limit on automations. Why your second retry may not be firing now: - Reactive triggers wait out their “within” (debounce) window after they fire. If your second crash happens during that window, the trigger won’t fire again yet. What to set to allow repeated retries: - Trigger posture: Reactive - Events to expect:
prefect.flow-run.Crashed
- Threshold: 1 - Within (debounce): 0s (or very small) - For each:
prefect.resource.id
(so each new run has its own bucket and can trigger again) That setup will fire on every crash and can keep resubmitting. Important guardrails (to avoid infinite loops): - There’s no native “max firings” on automations. If you want to stop after N attempts, add a simple guard: - Pass an
attempt
parameter in the automation’s Run Deployment action and increment it each time; have your flow read it and stop resubmitting after N. - Or add a second automation that alerts/escalates if there are ≥ N crashes within W minutes for that deployment. - Concurrency limits can also help prevent piling up retries. Notes on retries vs infra failures: - Flow-level retries won’t help if the run never starts (e.g., Fargate can’t pull the image). Your automation approach is the right tool for infra-startup failures. If you share a screenshot or JSON of your current automation, I can suggest the exact tweaks. References: - Automations guide - Debouncing (the within window) - Flow retries