Hey Prefect team, A dev on other team brought this...
# ask-community
d
Hey Prefect team, A dev on other team brought this to my attention: a subflow is stuck in running condition while the main is also stuck in cancelling . Interestingly enough, the subflows in this flow kick off spark compute jobs in another part of our ecosystem which did complete successfully. I have found that on our k8s cluster there are no running pods for either the flow or subflows, so it appears to be Prefect Cloud UI issue. Could you direct me as to how I can clear this out and potentially prevent this kind of thing happening in the future?
m
You can delete your flow runs in order to clear out the jobs stuck in 'cancelling', using the Python SDK or Rest API. Going forward you might find this example helpful, where you can filter your flow runs by bad states, and then delete those which are stuck in "cancelling". https://github.com/EmilRex/prefect-api-examples/blob/main/flow_runs.py It's possible that the same worker is submitting runs for execution immediately after a successful completion, which is what leads to certain runs being stuck in the 'running' state. Which versions of Prefect/prefect-kubernetes are you using? I'd suggest upgrading your prefect-kubernetes version to 0.3.7 if you haven't yet.
🙌 1