When a Flow loses the heartbeat and then is restarted (No heartbeat detected from the remote task; r...
j
Jessica Smith
06/29/2022, 8:42 PMWhen a Flow loses the heartbeat and then is restarted (No heartbeat detected from the remote task; retrying the run.This will be retry 2 of 3.) is there a flow State provided when the process starts again? Like a Retrying state or a Running state?
✅ 1
Jessica Smith
06/29/2022, 9:46 PMI see that in Server it sets it to Scheduled. But if that was happening in Cloud, my flow state handler should be able to log that. The state handler doesn't seem to be called at all, which makes me think it is submitted with a Running state
Copy code
await prefect.api.states.set_flow_run_state( flow_run_id=fr.id, state=Scheduled(message="Rescheduled by a Lazarus process."), )k
Kevin Kho
06/30/2022, 8:46 AM
The state handler happens when the flow starts, but the restart normally indicates that the Flow process could not even start (like Docker/Kubernetes not spinning up). So this is not something that can be acted upon in a state handler
a
Anna Geller
06/30/2022, 11:50 AM
I wouldn't dive as deep into the heartbeat issues, we don't have a reliable solution for this in 1.0 unfortunately and we will investigate after GA how we may tackle this in 2.0
j
Jessica Smith
06/30/2022, 1:28 PMSo let me go a bit wider and see if you guys can give me some advice. I am using the state handler to load some data into an object when the flow starts. The data is used across various pieces of my flows, it is very similar to the Prefect Context object.
Everything works fine when a flow runs without heartbeat issues, the data gets loaded and the tasks that rely on it succeed. But when the flow is restarted by Lazarus, the data does not get loaded and the tasks fail. When this happens I can use the Restart button on the failed flow to kick it off, and when it is restarted that way, the flow handler step works properly again.
If a state handler isn't reliable when a flow restarts from Lazarus, is there another method I could use to always run a bit of code (not in a Task) when a flow starts running, regardless of when/how it is started?
Jessica Smith
06/30/2022, 1:29 PMI'm going to look into caching the data, but I am not sure about that, so any recommendations would be appreciated
a
Anna Geller
06/30/2022, 1:49 PM
my recommendation would be to pass that data that should be shared across tasks as data dependency i.e. one task returns that and passes it to other tasks that need it
state handler is more to react to state changes e.g. if a flow run moves to a Failed state, perform some action; it's not meant to be used as a stateful component - perhaps the name of the feature confused you 😄 it's not to manage state but to manage state transition and perform actions on state transition
j
Jessica Smith
06/30/2022, 1:51 PMfair enough, I might go back and look at changing my structure around. Appreciate the info 👍
🙌 1
👍 1
13 Views
Prefect Community
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by