I know that you can set retries on individual task failures. Can you set a retry on a "flow of flows" if one flow fails?

It’s hard because retrying a

create_flow_run

with the same idempotency key just returns the result of what was already run. So you need to restart the Flow in the UI on the subflow, and then restart the main Flow. We don’t have restart through the API, only the UI has it. What you can do though is create a new flow run with a new idempotency key and then rely on caching so that you don’t repeat work that is already done

Is there an example of how that would work?

Man it’s been a while I don’t think so but there are three parts. First is the idempotency key, you want to fill it with a different value to trigger another flow run

with Flow(..) as flow:
    create_flow_run(..., idempotency_key=datetime.datetime.now())

Second is that you actually want this value to change from run to run instead of being fixed at build time so you need script-based storage (Github, etc). and then on the subflow side, apply

cache_for

and

cache_validator

to the expensive tasks so they don’t rerun

@task(cache_for=...)
def myfunc(x):
    return x + 1

Okay this is very helpful! Would we have to manually restart the flow run?

Still yes in both cases. idempotency just helps create a new child flow run because using the same idempotency key does nothing

Gotcha, okay this is great man thank you!~

How many subflows does your main flow have?

running 4 flows via orchestration

ok yeah that’s not too bad. i know a guy who had 20

haha well maybe waiting for prefect 2.0 is the move