https://prefect.io logo
Join Slack
Powered by
Our team wrote a flow state handler that changes t...
# prefect-community
k
Our team wrote a flow state handler that changes the new_state of the flow to Finished('message'). For some reason this causes our flows to look like they are running ad infinitum. For example The flows We ran a week ago show run duration of 7 days. But there isnt anything running as far as we can tell. Any ideas why changing the state to Finished State causes the flow runs in prefect UI to look like they are running forever? When I change our philosophy to setting the new state to Cancelled('message") we dont get see this behavior. The flow duration is exactly what it should be in that case. Just when we try to set the new state to a Finished instance.
k
This is Prefect 1 right? Could you try a 
Success
state instead?
k
This is prefect 1.
But our objective was to change from fail to Finished so we can distinguish from the dashboard directly which ones we forced into a Finished state.
k
I don’t think the UI has a color dedicated to that. Let me take a look
I can’t find it immediately so I need to ask the UI team and I’ll get back to you if this is possible
k
Cool Cool. It happens that a flow will fail and when I go look into it, I recognize it is due to some other team not sending their data to us the right way or something else like that. So what we have been doing is setting those flow runs states to Finished, which changes those flow runs to blue color in the UI, so we can distinguish those flows in the future.
k
Oh you are right there is a Finished state and it’s Blue in the UI.
I would need to try this to see if there is something wrong where the Flow doesnt termiante
k
Thank you! Yeah maybe a simple flow with a simple state handler that just converts the new state to Finished('any message here'). Our flows are hosted in prefect cloud in case that matters.
k
I tried this and I can see how it would be problematic adding this state handler to the Flow and if it changes to the state while there are still tasks ongoing. Did all your tasks Finish when you tried this?
Ok this works for me and it’s rendered as Blue:
Copy code
from prefect import Flow, task
from prefect.engine.state import Finished


def handler(obj, old_state, new_state):
    if isinstance(new_state, Finished):
        return Finished("finished")
    else:
        return new_state

@task()
def mytask():
    return 1

with Flow(name="POC2", state_handlers=[handler]) as flow:
    mytask()
k
Hey Kevin, Interesting. We have it similar. Essentially the only difference I can think of is that our handler logic goes more like
Copy code
if new_state.is_failed():
        return Finished("finished")
    else:
        return new_state
which after checking the codebase in prefect github looks like is_failed pretty much returns isinstance(new_state, Failed) and Failed is a child class of Finished I believe. We do get our color to change to blue in prefect UI, but our flows duration just wasnt reflecting that the flow was indeed finished. For example, we get the blue flow run state after 1 minute for our test flow runs but tomorrow when I log on I see the flow duration is 24 hours and tomorrow when I check the same flow run it says 48 hours. Are you not seeing the same behavior?
k
With the Flow above, it seems to end for me. Let me check the duration of that Flow that completed.
k
Thanks again for cross checking Kevin -- Really appreciated! Also, when We return Cancelled('finished') in the state handler using all the same code we are not seeing that same behavior with the duration increasing and increasing over time.
k
Ah I see you are right. The end date in wonky and now that I opened it, it does say duration was 7 hours. I will think a bit too see if we can do something immediate, otherwise, we may just have to open an issue on Github.
k
Interesting. I will think on it too. I'll check this thread when I log on tomorrow morning. Thanks again!
k
Yeah this seems a bit too deep to be worked around. I opened an issue here . Feel free to chime in
k
Much appreciated!
3 Views
Open in Slack
PreviousNext
Powered by