https://prefect.io logo
Title
c

Charles Leung

04/19/2023, 10:47 PM
Hi Prefect, is there a way to do retries from the point of failure? ie if the flow kicks off taskA and taskB runs, but taskB fails, how do we retry taskB only?
1
z

Zanie

04/20/2023, 2:57 AM
Hey! That’s how flow run retries work by default. We will not rerun previous tasks that completed successfully.
c

Charles Leung

04/20/2023, 5:16 PM
Thanks @Zanie, thats what I read as well. But for some reason when I tested this theory with a deployment that includes two tasks A and B (the second one, B, intentionally raises an error). When the process failed, I pressed rerun but somehow it still executed the previous task A before running task B. Code below for reference:
from prefect import flow, task
from prefect_shell import ShellOperation

@task()
def secondary():
    raise ValueError("Failed")

@task()
def primary():
    print("This is the primary task")
    return

@flow()
def main():
    primary()
    secondary()


if __name__ == "__main__":
    main()
z

Zanie

04/20/2023, 5:21 PM
You need to turn on result persistence for the task to be cached
i.e.
@task(persist_result=True)
We do this automatically if you turn on flow run retries, but if you just retry it after the fact we can’t know to persist them in the previous run
c

Charles Leung

04/20/2023, 9:51 PM
Thanks @Zanie. Im still not getting the expected output though.
from prefect import flow, task, logging, get_run_logger
from prefect_shell import ShellOperation

@task(persist_result=False)
def secondary():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Secondary Task")
    print("This is the secondary task")    
    raise ValueError("Failed")
    return

@task(persist_result=False)
def primary():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Primary Task")
    print("This is the primary task")

@flow(persist_result=False)
def main():
    primary()
    secondary()


if __name__ == "__main__":
    main()
In this scenario, I tried testing it with a deployment and expected "Primary Task" AND "Secondary Task" to be logged in the output on retry, since persist_result is False. But when I pressed retry in the deployment, i didnt get any of the logs even though i put persist_result=False
Sinilarly, I switched persist_result to be True, and I expected it to ONLY log "Secondary Task", since its the secondary() task that errored out in the first run, so it should skip the Primary task. However, it didnt log anything either. Am I missing something?
z

Zanie

04/20/2023, 10:25 PM
Hm I'm a bit confused :) let me give it a try tomorrow and get back to you if you haven't sorted it out
🙏 1
d

Deceivious

04/21/2023, 8:20 AM
-following this thread- Ive found retries to be iffy at times. Failed flows when retried usually "retries" but failes again with the same error . Creating a new flow run with the same inputs works as retry for me.
c

Charles Leung

04/21/2023, 4:03 PM
Yeah that makes sense. The reason why it's important for us is because our process takes hours so if it fails we want to make sure it is retried from that point of failure. If we recreate a new flow run with the same inputs it will start from the beginning
z

Zanie

04/21/2023, 4:05 PM
You can use cache keys if you generally want to avoid recomputing tasks.
This should work however. If it's not working as intended, we'll fix it asap
If someone could open an issue with a minimal example that'd be helpful. I won't be able to get to it today.
c

Charles Leung

04/21/2023, 4:56 PM
Issue created - https://github.com/PrefectHQ/prefect/issues/9292 I can follow up with you here if theres any/no updates too. Thanks all!
z

Zanie

04/21/2023, 5:22 PM
It looks like you deleted the rest of your report by accident 🙂
Can you include the persist_result=True behavior too? I’d expect it to fail with False
c

Charles Leung

04/21/2023, 5:41 PM
Sorry! I copied the rest of the report back in, and the result of persist_result=True gave the same result (no output)
Is the behavior normal with False? I thought it was supposed to rety from the beginning so at least the logs are there
z

Zanie

04/21/2023, 6:03 PM
If the results are not persisted you’ll get a
MissingResult
error when you access the result of the task — you can see this happening for primary-0 as it retries from the beginning
After https://github.com/PrefectHQ/prefect/pull/9152 is done it will run again instead
❤️ 1
c

Charles Leung

04/21/2023, 7:27 PM
Ah perfect thanks!