What's a good practice for handling subflow failur...
# ask-community
k
What's a good practice for handling subflow failure? For e.g. I have a case of a flow triggering many (e.g. 15) subflows and one subflow fails.
d
ive set
return_state
to True. After all the sub flow runs are over, I have codes for exception handling over the sub flows. But you are going to be more specific about your use case.
k
ive set return_state to True.
Yeah, I do this as well and fetch the result. What's happening is that I have many upserts to do at once to a data warehouse. Each table upsert is one subflow. It would be great if the failed subflow could store the result somewhere such as to an Artifact maybe? Not sure yet. The issue is that it's $ costly to redo the upsert process each time so I'm trying to figure out a way to get the subflow failures and persist those somewhere so that subflow could be retriggered.
d
Not entirely sure about your usecase. But you could break down the sub flow into multiple task and
cache
your task result?
Additionally you can also cache your sub flow results so that successful "upserts" are not performed twice.
k
Caching the subflow results would be like so I guess? https://docs.prefect.io/latest/concepts/results/#caching-of-results-in-memory
Not in the memory - but on some remote storage infrastructure. Multiple flowrun [main flow not the subflows] have different memory space.
k
I see, thanks so much!