<@ULVA73B9P> Let's assume that flow, which is runn...
# ask-marvin
r
@Marvin Let's assume that flow, which is running on local worker against on-prem Prefect has setuped : retries=5, retry_delay_seconds=60 I noticed, that when flow is started, new process is forked, which is alive during flow run and also during waiting for retry and during retries. I would prefer to finish process after failure and start new one during retry. Is it possible to achieve this ?
m
thought for 279.5 seconds
Yes — what you’re seeing is expected for flow-level retries in Prefect 3.x. Short answer: No, not with built-in flow retries. If you set
retries=5
and
retry_delay_seconds=60
on the flow, Prefect treats those as in-process retries. The same spawned process stays alive, waits, and then re-runs the flow in that same process. A few important details: - The worker creates the process/container/job once for that flow run - If the flow fails but still has retries left, Prefect moves it to
AwaitingRetry
- The same execution environment handles the next attempt - Workers do not launch a brand new process for each retry attempt This behavior is reflected in recent Prefect issues/PRs around
retry_type='in_process'
and `AwaitingRetry`: - PR #21933 - Issue #17913 So if your goal is: failed attempt ends process -> retry starts fresh process that is not how flow retries work today. What you can do instead 1. Remove flow-level retries Let the flow fail outright. 2. Trigger a new flow run for each attempt That gives you a fresh process each time, because each retry is now a new run, not a retry within the same run. Common patterns: - an outer orchestrator/system re-submits the deployment - another Prefect flow submits the child deployment/run and decides whether to try again - external tooling (cron, Airflow, CI, custom script) re-triggers on failure Conceptually:
Copy code
text
attempt 1 = flow run A -> process A
attempt 2 = flow run B -> process B
attempt 3 = flow run C -> process C
instead of:
Copy code
text
flow run A -> process A -> retry inside process A -> retry inside process A
When to choose which - Use flow retries if you want simple retry semantics and can tolerate reusing the same process - Use new flow runs per attempt if you need a truly clean interpreter/process/environment every time If you want, I can show you a Prefect 3 pattern for fresh-process retries using: - a parent flow that resubmits a deployment, or - a small external script that retries
prefect deployment run
/ flow submission and I can tailor it to your setup (local worker + on-prem server).