To answer the question though, Flows update their state by hitting the Prefect API. If the compute quite literally dies, there is no update being sent. So if a pod literally crashes, there is no compute that can hit the API and update the Flow. But I think in general we have a related situation we are aware of that is on the roadmap to be fixed. If you cancel your Flow from the UI, you can get pods or a cluster left turned on because the exit is not graceful. In this case, we may need something on the agent that is responsible for handling the shutdown. I think in this situation, that same mechanism may be able to be used to change the state (not 100% sure). I think what we should do here is create a feature request and reference that other issue as they may be related. Will take care of it. In the meantime though, I am not sure there is anything to help you. If you are on Cloud though, you might be able to turn off Version Locking for the Flow. In Server, we actually have complaints that work is duplicated (which I think is more expected, especially with Dask) because the work is restarted. Version Locking in Cloud stops it from re-running

What happens now is more like this: resource manager: Hey prefect flow run, I know you're here, but I need your CPU and memory resources, so I'm gonna terminate you in a bit, but don't worry, I'm gonna move you to a different machine. (sends SIGTERM signal) prefect flow run: just dies, doesn't notify the API at all resource manager: starts another prefect flow run on a different machine prefect flow run: Hey this is confusing, the API says the task is running but I'm not running anything, better kill myself. (dies)

Issue is here: https://github.com/PrefectHQ/prefect/issues/5198

Feel free to add details (if you want to copy paste your description that you outlined here)