Adam Brusselback

    Adam Brusselback

    11 months ago
    I have a flow that has 4 tasks, first being a parameter, second returning an intermediary result, third using that result to get some more intermediary results < fails here, fourth using all these intermediary results to run a query against my database. In testing I caused some failures on the 3rd step, and when trying to restart the failed job the intermediary results from the second step are set as None
    If I just start a new run, everything is fine. It's just restarting the failed run that has problems.
    Kevin Kho

    Kevin Kho

    11 months ago
    What results do you have set on these tasks?
    Adam Brusselback

    Adam Brusselback

    11 months ago
    Just a simple string (which is a key I use to look up a dict of secrets in my next task)
    Kevin Kho

    Kevin Kho

    11 months ago
    If you are restarting a flow run, the results are loaded in and then passed to the next task so that you don’t need to re-run the task
    Adam Brusselback

    Adam Brusselback

    11 months ago
    Ah that's my problem, I need to instrument my task with a result type (prefect result should work just fine in this case). I was just surprised it wasn't rerunning all tasks if the result wasn't persisted
    On a related note, if I have a task where I don't want to persist the results but I just want it re-run in the event of a flow re-run, is there a way to do that?
    Kevin Kho

    Kevin Kho

    11 months ago
    Kind of. You can set the state to Failed or maybe Scheduled through the API or UI.
    Adam Brusselback

    Adam Brusselback

    11 months ago
    It actually looks like I want to use SecretResult here, as it ensures the code is re-run and I am writing my own code to get my secrets and I don't want it persisted anywhere. So I am changing my task to subclass the SecretBase rather than using the @task annotation
    Very much appreciate you pointing me in the right direction. Hope I haven't been too much of a pain with these questions over the past weeks.
    Kevin Kho

    Kevin Kho

    11 months ago
    Oh that’s news to me. Of course no problem!