Channels
#prefect-community
Title
# prefect-community
k
Kyle Austin
03/23/2022, 2:34 PMOur team wrote a flow state handler that changes the new_state of the flow to Finished('message'). For some reason this causes our flows to look like they are running ad infinitum. For example The flows We ran a week ago show run duration of 7 days. But there isnt anything running as far as we can tell. Any ideas why changing the state to Finished State causes the flow runs in prefect UI to look like they are running forever? When I change our philosophy to setting the new state to Cancelled('message") we dont get see this behavior. The flow duration is exactly what it should be in that case. Just when we try to set the new state to a Finished instance.
k
Kevin Kho
03/23/2022, 2:35 PM
This is Prefect 1 right? Could you try a
Successk
Kyle Austin
03/23/2022, 5:51 PMThis is prefect 1.
But our objective was to change from fail to Finished so we can distinguish from the dashboard directly which ones we forced into a Finished state.
k
Kevin Kho
03/23/2022, 5:53 PM
I don’t think the UI has a color dedicated to that. Let me take a look
I can’t find it immediately so I need to ask the UI team and I’ll get back to you if this is possible
k
Kyle Austin
03/23/2022, 6:01 PMCool Cool. It happens that a flow will fail and when I go look into it, I recognize it is due to some other team not sending their data to us the right way or something else like that. So what we have been doing is setting those flow runs states to Finished, which changes those flow runs to blue color in the UI, so we can distinguish those flows in the future.
k
Kevin Kho
03/23/2022, 6:02 PM
Oh you are right there is a Finished state and it’s Blue in the UI.
I would need to try this to see if there is something wrong where the Flow doesnt termiante
k
Kyle Austin
03/23/2022, 6:04 PMThank you! Yeah maybe a simple flow with a simple state handler that just converts the new state to Finished('any message here'). Our flows are hosted in prefect cloud in case that matters.
k
Kevin Kho
03/23/2022, 6:17 PM
I tried this and I can see how it would be problematic adding this state handler to the Flow and if it changes to the state while there are still tasks ongoing. Did all your tasks Finish when you tried this?
Ok this works for me and it’s rendered as Blue:
Copy code
from prefect import Flow, task
from prefect.engine.state import Finished
def handler(obj, old_state, new_state):
if isinstance(new_state, Finished):
return Finished("finished")
else:
return new_state
@task()
def mytask():
return 1
with Flow(name="POC2", state_handlers=[handler]) as flow:
mytask()k
Kyle Austin
03/24/2022, 1:20 AMHey Kevin, Interesting. We have it similar. Essentially the only difference I can think of is that our handler logic goes more like
Copy code
if new_state.is_failed():
return Finished("finished")
else:
return new_statek
Kevin Kho
03/24/2022, 1:29 AM
With the Flow above, it seems to end for me. Let me check the duration of that Flow that completed.
k
Kyle Austin
03/24/2022, 1:30 AMThanks again for cross checking Kevin -- Really appreciated!
Also, when We return Cancelled('finished') in the state handler using all the same code we are not seeing that same behavior with the duration increasing and increasing over time.
k
Kevin Kho
03/24/2022, 1:31 AM
Ah I see you are right. The end date in wonky and now that I opened it, it does say duration was 7 hours. I will think a bit too see if we can do something immediate, otherwise, we may just have to open an issue on Github.
k
Kyle Austin
03/24/2022, 1:34 AMInteresting. I will think on it too. I'll check this thread when I log on tomorrow morning. Thanks again!
k
Kevin Kho
03/24/2022, 1:39 AM
Yeah this seems a bit too deep to be worked around. I opened an issue here . Feel free to chime in
k
Kyle Austin
03/24/2022, 1:40 AMMuch appreciated!
3 Views