For a given flow, if a task fails I am wanting to ...
# ask-community
t
For a given flow, if a task fails I am wanting to write the state to my local machine, and be able to load it back up later. I am looking into state handlers and persisting output using
LocalResult
but I'm wondering if there are Prefect features that are particularly appropriate for this use case.
đź‘€ 1
d
Hey @tkanas! When you say “state” here, can you elaborate a bit? Do you want to write the result of previous tasks, the flow context, the state object for that task run?
To cache results, take a look here: https://docs.prefect.io/core/concepts/persistence.html
t
That's something we are still thinking about. Basically the state of the flow just before failure, so that a user could possibly load it back up and inspect it later.
d
Hmmm, but this is in response to failure?
So you want to catch all exceptions, write something down, and then fail the task run?
t
I think so. I'm still thinking about what the "something" is and what the best way is of "writing it down". So far I have been looking at using a
state_handler
(that would check
state.is_failed()
and then dispatch to the thing that writes the state) in conjunction with a persisting method as described in the link you provided above. I just haven't figured out the best way of doing it.
d
So I think the key point here is that you want to know the state “before” the failure
I’m not sure a state handler will help with that, since they fire on state change
However, have you tried setting up Prefect Server or Prefect Cloud? If you have an orchestration layer you would be able to see the state of the failed flow along with logs
Or do you want to “load” a certain state of the world into a python interpreter locally?
Another way of asking my question: are you having trouble identifying an issue inside a task, or testing a solution?
t
Saving and loading a state of the world sounds like it might fit the bill. Preferably the state right before failure. If that's not possible, then maybe the state right after failure?
How would you save and load a "state of the world"?
d
Locally, I’m not sure we have a way to do this
But if you’re using Prefect Server/Cloud, you can manipulate task run states and flow run states with the graphql API
that would allow you to watch a flow run in a given state of the world
t
OK, I will look into that. Thanks!
d
Anytime!