Thread
#prefect-community
    Jason Motley

    Jason Motley

    6 months ago
    I know that you can set retries on individual task failures. Can you set a retry on a "flow of flows" if one flow fails?
    Kevin Kho

    Kevin Kho

    6 months ago
    It’s hard because retrying a
    create_flow_run
    with the same idempotency key just returns the result of what was already run. So you need to restart the Flow in the UI on the subflow, and then restart the main Flow. We don’t have restart through the API, only the UI has it. What you can do though is create a new flow run with a new idempotency key and then rely on caching so that you don’t repeat work that is already done
    Jason Motley

    Jason Motley

    6 months ago
    Is there an example of how that would work?
    Kevin Kho

    Kevin Kho

    6 months ago
    Man it’s been a while I don’t think so but there are three parts. First is the idempotency key, you want to fill it with a different value to trigger another flow run
    with Flow(..) as flow:
        create_flow_run(..., idempotency_key=datetime.datetime.now())
    Second is that you actually want this value to change from run to run instead of being fixed at build time so you need script-based storage (Github, etc). and then on the subflow side, apply
    cache_for
    and
    cache_validator
    to the expensive tasks so they don’t rerun
    @task(cache_for=...)
    def myfunc(x):
        return x + 1
    All this will create a new flow run when you restart but respect the cached work. Or you can click Restart in the UI for the subflow, then click it for the main flow and that should work
    Jason Motley

    Jason Motley

    6 months ago
    Okay this is very helpful! Would we have to manually restart the flow run?
    Or does the indempendency do that?
    Kevin Kho

    Kevin Kho

    6 months ago
    Still yes in both cases. idempotency just helps create a new child flow run because using the same idempotency key does nothing
    And this is all much improved in Prefect 2.0 one way or another, especially retrying subflows
    Jason Motley

    Jason Motley

    6 months ago
    Gotcha, okay this is great man thank you!~
    Kevin Kho

    Kevin Kho

    6 months ago
    How many subflows does your main flow have?
    Jason Motley

    Jason Motley

    6 months ago
    running 4 flows via orchestration
    Kevin Kho

    Kevin Kho

    6 months ago
    ok yeah that’s not too bad. i know a guy who had 20
    and i know this is painful
    Jason Motley

    Jason Motley

    6 months ago
    haha well maybe waiting for prefect 2.0 is the move