Hello, I have a flow that creates other subflows. ...
# ask-community
d
Hello, I have a flow that creates other subflows. During one of the flow runs, 3 subflows were spawned and 1 of them failed causing the "main" flow to enter FAILED state. Is it possible to retry the failed flow with the same parameters and bring the "main" flow run state to SUCCESS ? Sample example in image below.
t
You could try to start the sub-flow from within a task with task-retries active.
d
Yes that is always possible. But I wanted to know if the flow states can be fixed after the failure has already occurred.
Just for an example, The main flow triggers subflow for each website from a dynamic list of websistes and the subflows task is to take a snapshot of the webpage and save the SS to s3. One of the sites had an outage at that time and failed. I wanna try attempt to get that SS maybe 6 hours later? Just an example, of course, the real issue is more complex 😄 .
c
You’re asking if main flow fails, and you retry a sub-flow, if the main flow is marked as success AFTERwards? no - you would need to re-run the main flow to success
✅ 1
d
Do you think this could be a new feature request? Provided that the subflows do not return any data, add support to retry failed subflows that overwrites previous faile subflow and resume main flow ? Unsure how to handle such cases.
Guess what I am looking for is something similar to github workflow's "Retry all failed workflows" feature but on a subflow level ; if that makes any sense 😅
c
what is the end goal in doing it this way, versus cacheing / persisting results, and retrying the main flow?
d
Yes if caching is involved i guess retrying main flow would not incur the processing cost of each sub flow. But, I was additionally thinking about using each flow run info via prefect api and generating daily reports (Assuming each unique "main" flow id maps with daily scheduled process). Also each failed flow could be an
incident
that represents data issues on a particular day that could be marked as
resolved
when the retry causes the flow run to be in
SUCCESS
state. But yeah now I kinda feel like I am trying to use prefect for something it was not originally intended to be used for.
a
I also would want retrying to work like this @Deceivious, this would closely resemble the way Airflow works which I'm used to. As a prefect newbie I'm trying to wrap my head around the different framework around retries and flow states
d
Sounds like a lot of work but maybe worth requesting for a feature request
a
I think it's an intentional design decision, just a different paradigm