Hi Prefect, is there a way to do retries from the ...
# ask-community
c
Hi Prefect, is there a way to do retries from the point of failure? ie if the flow kicks off taskA and taskB runs, but taskB fails, how do we retry taskB only?
1
z
Hey! That’s how flow run retries work by default. We will not rerun previous tasks that completed successfully.
c
Thanks @Zanie, thats what I read as well. But for some reason when I tested this theory with a deployment that includes two tasks A and B (the second one, B, intentionally raises an error). When the process failed, I pressed rerun but somehow it still executed the previous task A before running task B. Code below for reference:
Copy code
from prefect import flow, task
from prefect_shell import ShellOperation

@task()
def secondary():
    raise ValueError("Failed")

@task()
def primary():
    print("This is the primary task")
    return

@flow()
def main():
    primary()
    secondary()


if __name__ == "__main__":
    main()
z
You need to turn on result persistence for the task to be cached
i.e.
@task(persist_result=True)
We do this automatically if you turn on flow run retries, but if you just retry it after the fact we can’t know to persist them in the previous run
c
Thanks @Zanie. Im still not getting the expected output though.
Copy code
from prefect import flow, task, logging, get_run_logger
from prefect_shell import ShellOperation

@task(persist_result=False)
def secondary():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Secondary Task")
    print("This is the secondary task")    
    raise ValueError("Failed")
    return

@task(persist_result=False)
def primary():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Primary Task")
    print("This is the primary task")

@flow(persist_result=False)
def main():
    primary()
    secondary()


if __name__ == "__main__":
    main()
In this scenario, I tried testing it with a deployment and expected "Primary Task" AND "Secondary Task" to be logged in the output on retry, since persist_result is False. But when I pressed retry in the deployment, i didnt get any of the logs even though i put persist_result=False
Sinilarly, I switched persist_result to be True, and I expected it to ONLY log "Secondary Task", since its the secondary() task that errored out in the first run, so it should skip the Primary task. However, it didnt log anything either. Am I missing something?
z
Hm I'm a bit confused :) let me give it a try tomorrow and get back to you if you haven't sorted it out
🙏 1
d
-following this thread- Ive found retries to be iffy at times. Failed flows when retried usually "retries" but failes again with the same error . Creating a new flow run with the same inputs works as retry for me.
c
Yeah that makes sense. The reason why it's important for us is because our process takes hours so if it fails we want to make sure it is retried from that point of failure. If we recreate a new flow run with the same inputs it will start from the beginning
z
You can use cache keys if you generally want to avoid recomputing tasks.
This should work however. If it's not working as intended, we'll fix it asap
If someone could open an issue with a minimal example that'd be helpful. I won't be able to get to it today.
c
Issue created - https://github.com/PrefectHQ/prefect/issues/9292 I can follow up with you here if theres any/no updates too. Thanks all!
z
It looks like you deleted the rest of your report by accident 🙂
Can you include the persist_result=True behavior too? I’d expect it to fail with False
c
Sorry! I copied the rest of the report back in, and the result of persist_result=True gave the same result (no output)
Is the behavior normal with False? I thought it was supposed to rety from the beginning so at least the logs are there
z
If the results are not persisted you’ll get a
MissingResult
error when you access the result of the task — you can see this happening for primary-0 as it retries from the beginning
After https://github.com/PrefectHQ/prefect/pull/9152 is done it will run again instead
❤️ 1
c
Ah perfect thanks!